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SUBJECT: 


Pershing  II  Follow-On  Test: 
Analysis 


Size  Reduced  by  Sequential 


1 


By  memorandum  of  30  August  1982  (Reference  1),  the  Under 
Secretary  of  the  Army  tasked  the  service  to  "review  our 
[operational  test]  methodology,  to  include  considerations  of 
mathematical  rigor,  risks,  planning  horizon,  costs,  and 
operational  matters."  In  discussion  of  this  matter  with  the 
author,  he  further  elaborated  the  objectives: 

a)  Minimize  cost  of  testing  over  the  program  life.  Monitor 
all  test  results,  including  those  of  components  as  well  as  of  the 
system,  to  minimize  "no-tests"  and  to  save  on  full-up  tests.  Use 
sequential  analysis  to  further  pare  requirements  for  missile 
flights . 

b)  Criteria  of  test  adequacy  should  be  no  more  severe  than 
those  of  other  services  (e.g.,  Minuteman,  Poseidon). 

c)  Challenge  the  necessity  for  an  annual  update. 

d)  Consider  whether  testing,  maintenance  float,  and  reload 
were  independent  requirements  as  opposed  to  multiple  missions  for 
the  same  inventory  of  missiles. 

The  task  was  passed  to  the  Army  Research  Office  (Research 
Triangle,  NC)  which  manages  the  business  of  the  Army's  Mathematics 
Steering  Committee  (Dr.  Jagdish  Chandra,  Chairman),  supporting 
mathematical  research  of  relevance  to  the  Army  and  the 
improvements  in  mathematical  methods  employed  in  the  Army's 
research  and  study  agencies. 

The  work  summarized  here  is  composed  of  contributions  of 
several  statisticians  whose  aid  was  solicited  by  the  AMSC:  Dr. 
Michael  Woodroofe  (University  of  Michigan)*,  Dr.  Nozer 
Singpurwalla  (George  Washington  University),  and  Dr.  Robert  Launer 
(Army  Research  Office),  as  well  as  the  author  of  this  report. 
Others  have  provided  informal  comments  and  criticisms.  An  early 
version  of  this  paper,  prior  to  the  author's  knowledge  of  this 
other  research,  was  presented  as  a  talk  at  a  conference  of  Army 
mathematicians  (Reference  2). 


*  At  Rutgers  University  during  the  course  of  this  research. 


Chapter  I 
The  Problem 


Two  documents  combined  set  forth  the  guidance  the  Joint 
Chiefs  of  Staff  have  provided  to  the  military  services  regarding 
the  conduct  and  reporting  of  tests  of  certain  systems.  For  the 
Army  only  the  Pershing  Missile  system  is  covered  (Pershing  I  and 
la,  and  now  Pershing  II). 

In  a  memorandum  of  1975  (Reference  3),  the  Joint  Chief  of 
Staff  directed  that  numerical  confidence  statements  should  be 
based  on  WSEG  Report  92C  (Reference  4),  an  extract  of  which  is  at 
Appendix  C.  "The  goal  of  a  test  program  should  be  to  allow 
detection  of  a  minimum  change  of  X  percent  at  the  Y  percent 
confidence  level."  *  It  suggests,  by  way  of  example,  the  use  of 
Fisher's  Exact  Test  to  demonstrate  success  or  failure  in  meeting 
this  criterion. 

Refeiences  3  and  4  have  just  been  superseded.  The  revisions 
(References  5  and  6)  eliminate  an  ambiguity  and  add  considerations 
not  previously  called  for  and  not  discussed  here  except  to  note 
that  the  criteria  to  be  applied  to  Pershing  II  are  now  less 
demanding  than  those  applied’  to  strategic  systems.  Fisher's  Exact 
Test  is  still  countenanced. 

This  use  of  this  criterion  appeared  to  the  author  to  lack  a 
sound  statistical  justification,  and  attempts  to  patch  it  up  were 
unsuccessful.  Appeal  to  a  number  of  practicing  statisticians 
within  and  outside  the  Army  supported  my  challenge  to  Fisher's 
Exact  Test  (FET)  in  its  application  to  Pershing  reliability 
tracking.  No  one  was  contesting  the  ability  of  the  FET  to  provide 
estimates  of  the  probability  that  two  samples,  which  have  yielded 
pass-fail  data,  come  from  the  same  parent  population,  though 
Kendall  and  Stuart  (Reference  7),  do  condemn  its  use  for  small 
samples . 

With  such  an  error  apparently  arising  from  an  application  of 
the  methods  of  the  "frequency"  school  of  statistics,  the  obvious 
alternative  was  to  try  the  methods  of  the  "Bayesian"  school. 

There  are  many  expositions  of  methods  based  on  the  use  of 
Bayes'  Theorem,  the  most  recent  of  which — "Bayesian  Reliability 
Analysis"  by  Martz  and  Waller — (Reference  8)  I  shall  quote  at 
intervals.  Among  the  works  arguing  for  the  adoption  of  Bayesian 
methods,  the  following  are  noteworthy: 


Raiffa  and  Schlaifer  -  Applied  Statistical  Decision  Theory 
(Reference  9)  with  a  very  complete  description  of  the  method  of 
conjugate  prior  distributions. 

Jaynes  E.T.,  "Prior  Probabilities"  (IEEE  Transactions  on 
System  Science  and  Cybernetics,  September  1968)  (Reference  10). 
Deduction  from  the  principles  of  maximum  entropy  and  invariance 
under  certain  group  transformations  leads  directly  to  the  Beta 
distribution  as  conjugate  prior  to  a  Bernoulli  process;  indeed  to 

~  F*"'  o-s>  U 

where  s  is  the  number  of  successes  in  n  trials  observed  as  the 
basis  for  estimating  p.  This  removes  some  of  the  "ad  hoc"  or 
"mathematically  convenient"  color  of  conjugate  priors  when  relying 
on  Raiffa  and  Schlaifer. 

Martz  and  Waller  perhaps  epitomize  the  case  best: 

"There  are  several  benefits  in  using  Bayesian  methods  in 
reliability.  First  of  all,  it  is  important  to  recognize  that  all 
statistical  inferential  theories,  whether  sampling  theory, 
Bayesian,  likelihood,  or  otherwise,  require  some  degree  of 
subjectivity  in  their  use.  Sampling  theory  requires  assumptions 
concerning  such  things  as  a  sampling  model,  confidence 
coefficient,  which  estimator  to  use,  and  so  on.  For  example,  a 
sampling  theory  analysis  proceeds  as  if  it  were  believed  a  priori 
that  the  data  were  exactly  [exponentially]  distributed,  that  each 
observation  had  exactly  the  same  mean  life  0,  and  that  each 
observation  was  distributed  exactly  independently  of  every  other 
sample  observation.  The  Bayesian  method  provides  a  satisfactory 
way  of  explicitly  introducing  and  organizing  assumptions  regarding 
prior  knowledge  or  ignorance.  These  assumptions  lead  via  Bayes* 
theorem  to  posterior  inferences,  that  is,  inference  obtained  once 
the  data  have  been  incorporated  into  the  analysis,  about  the 
reliability  parameter(s)  of  interest.  Bayes*  theorem  provides  a 
simple,  error-free  truism  for  incorporating  the  prior  information. 
The  engineering  judgment  and  prior  knowledge  are  brought  out  into 
the  open  and  are  there  for  everyone  to  see  instead  of  being 
quietly  hidden.  The  engineer  usually  appreciates  this  opportunity 
to  divulge  such  prior  information  in  a  formalized  way." 

The  authors  I  commend  are  not,  on  philosophical  matters,  in 
complete  agreement,  and  the  authors  (and  critics)  of  the  methods 
proposed  in  this  paper  have  their  differences,  some  of  which 
become  important  as  we  proceed. 


Suffice  it  to  say  that  the  Bayesian  approach  requires  a  more 
careful  statement  of  the  problem,  to  include  in  particular  the 
prior  distribution  function,  costs  and  risks:  matters  which  the 
frequentists  collapse  into  the  confidence  limits  ^C.  and  (J  . 

If  there  is  indeed  a  legitimate  uncertainty  in  (the  form  of)  the 
prior  distribution,  that  uncertainty  must  surely  propagate  into  an 
uncertainty  in  the  predictions  for  the  process.  In  some  cases 
results  can  be  shown  to  be  insensitive  to  the  prior,  and  thus  a 
convergence  of  Bayesian  and  frequentist  answers  occurs;  but 
lacking  such  invariance,  the  frequentists  are  hard  pressed  to 
prove  they  have  solved  the  right  problem. 

Having  said  this,  T  must  confess  that  for  some  purposes  we 
shall  employ  the  frequentist  approach,  primarily  because  a  full 
Bayesian  solution  has  not  been  worked  out. 

Section  ^L.  Literal  Interpretation  of  JCS  Guidance: 

".  .  .  annual  .  .  .  detection  of  a  minimum  reliability  change 

of  X  percent  at  the  Y  percent  confidence  level." 

A  "change"  in  something  means  that  its  previous  value  has 
been  defined.  It  would  appear  that  an  evaluation  of  the  results 
of  the  first  year's  Follow-on-Test  (FOT)  is  to  be  compared  to  that 
of  the  Operational  Test  (the  base-line) (OT) ,  and  the  evaluations 
of  subsequent  FOTs  are  to  be  compared  to  the  evaluations  made  a 
year  ago.  The  tests  being  of  something  less  than  the  full  combat 
mode  of  the  system,  projection  to  combat  capability  is  to  be  made; 
thus  while  test  results  are  to  be  reported,  they  are  to  be 
interpreted  as  well.  This  interpretation  is  surely  to  be  based  on 
all  prior  knowledge  of  system  performance;  i.e.,  all  prior  testing 
as  well  as  that  most  recently  at  hand,  "weighted"  (one  might  say) 
by  expert  judgment  of  the  relevance  of  older  tests  and  analysis. 

In  the  case  of  Pershing  II,  we  shall  have  an  inventory  of 
missiles  produced  over  a  period  of  time  and  expected  to  be  in 
service  for  a  longer  period.  From  the  point  of  view  of 
homogeneity,  the  inventory  may  need  to  be  divided  into  two  or  more 
blocks,  based  on  the  significance  of  any  changes  in  the  production 
process  during  the  run.  When  they  are  subjected  to  (annual)  test, 
missiles  will  be  of  different  ages  as  well  from  different  blocks; 
so  serial  number  and  age  may  influence  reliability  at  the  time  of 
testing  or  use  in  combat.  It  is  clear,  then,  that  in  treating  of 
a  "change"  in  reliability,  we  are  dealing  with  an  uncertain  base. 
Options  which  are  open  to  us  include: 

a)  Computing  a  "best"  estimate  from  the  OT  firings,  and 
treating  it  as  the  exact  value  of  the  reliability  at  that  time  of 
all  the  inventory. 


b)  Computing  as  in  (a),  but  associating  an  uncertainty 
(standard  deviation)  to  it  also,  to  describe  the  uncertain 
reference  point. 

In  either  case,  the  results  of  each  subsequent  (annual)  test 
would  be  compared  to  this  as  standard. 

c)  Computing  as  in  (b),  but  then  modifying  the  estimates 
using  the  results  of  subsequent  tests  (more  trials,  more 
successes,  more  failures).  There  are  extremes  in  this  process 
which  are  to  be  avoided: 

(i)  This  modification  might  consist  of  using  only  the 
previous  year's  results  as  indication  of  the  remaining  inventory. 

(ii)  This  modification  might  consist  of  accumulating 
the  results  of  all  prior  tests,  without  regard  to  the  aging  effect 
or  block  modifications. 

Judgment  is  clearly  needed.  Limiting  the  criterion  to  the 
smallness  of  the  latest  annual  change  (with  small  samples  in  the 
two  cases)  could  result  in  a  dangerous  accumulation  of  change  over 
the  system  life.  On  the  other  hand,  where  no  statistically 
significant  change  has  been  detected,  it  would  be  reasonable  to 
add  one  year's  results  to  the  results  of  the  whole  prior  test 
series  of  a  homogeneous  block  in  estimating  the  average  value  at, 
say,  the  average  age  of  the  tested  articles.  It  is  probably  not 
possible  to  specify  in  advance  the  details  of  the  critical  results 
to  be  reported.  What  is  more  important  is  that  analyses  be 
conducted  to  discover  what  are  the  constant  and  what  are  the 
variable  components  of  the  system  reliability.  Finally,  detection 
of  a  trend  should  make  it  possible  to  forecast  when  the  results  of 
that  trend  will  no  longer  be  tolerable,  and  so  signal  the  degree 
of  urgency  with  which  management  should  act  to  correct  the  trend. 

d)  This  brings  us  to  the  question  of  the  frequency  of 
reporting  the  results  of  testing  and  analysis.  The  current 
practice  is  an  annual  report  which  probably  has  its  roots  in 
adminstrati ve  cycles.  The  technical  problems  which  reporting 
communicates  to  management  are  probably  of  two  sorts:  long-term 
aging  with  gradual  deterioration,  ("one-hoss  shay"  syndrome)  and 
catastrophic  failures.  The  latter  tend  to  announce  their  presence 
in  consistent  repetitions  of  particular  failure  modes,  and  so  call 
for  out-of-cycle  action  no  matter  what  the  standard  interval 
between  reports.  The  former,  on  the  other  hand,  are  evidence  of 
problems  only  slowly  exacerbating,  and  so  allow  a  more  leisurely 
pace  of  administrative  response.  Alternatives  to  the  present 
annual  cycle  are  proposed  below,  for  situations  in  which  no 
guarantee  of  a  clear  bright  green  light  or  red  light  is  available 
annually:  (i)  A  guarantee  can  be  given  of  a  low  likelihood  of 
having  to  wait  more  than,  say,  16  months  for  such  a  signal,  along 
with  the  provision  of  a  technical  review  of  all  failures  showing 
any  repetitions  of  mode.  (ii)  Administratively,  skipping  one 
year's  report  may  be  simpler. 


These  options  will  be  explored  in  one  or  more  places  in  the 
mathematical  sections  to  follow. 

Two  assumptions  have  immediately  to  be  disposed  of: 

1)  Because  Fisher's  Exact  Test  is  mentioned  in  JCS 
guidance,  its  use  is  correct,  and  mandatory. 

Fisher's  Exact  Test  is  an  enumeration  of  all  possible 
relative  outcomes  in  two  series  of  pass-fail  tests,  subject  to  the 
restraints  that  the  numbers  of  tests  in  each  series  be  fixed  and 
the  combined  number  of  successes  also.  It  yields  the  probability 
that  the  articles  tested  in  the  two  series  were  drawn  from  the 
same  population — one  with  a  fixed  probability  of  pass.  If  the 
total  number  of  successes  is  not  controlled,  the  results  of 
FET  admit  of  this  interpretation  only  in  the  limit  of  large 
samples.  Given  that  the  probability  of  success  could  be  different 
in  the  two  populations,  it  is  sometimes  claimed  that  FET  can  be 
used  to  estimate  the  probability  that  they  differ  by  prescribed 
amounts.  This  claim  is  unwarranted.  The  JCS  could  be  faulted  for 
suggesting  the  test,  but  they  did  not  underwrite  the  extended  use 
as  in  the  Army's  methodology.  (See  Kendall  and  Stuart;  also 
Chapter  III). 

2)  We  can  know  the  reliability  of  an  object. 

We  shall  never  know  the  "true"  as-manufactured  reliability  of 
the  components  of  the  Pershing  system,  and  much  of  such  knowledge 
as  we  do  gain  will  come  at  the  expense  of  tactical  inventory.  It 
may  be  that,  for  the  purposes  of  designing  tests  of  operational 
reliability,  we  need  not  know  this  a  priori  probability  with  any 
great  accuracy;  and  so  methods  which  treat  it  as  known  for  this 
purpose  may  be  satisfactory.  This  does  not  justify  the  assumption 
when  analyzing  the  results  of  actual  tests. 


Section  2.  Mathematical  Preliminaries 


Hayes'  Theorem:  The  Need  for  a  Prior  Distribution 

Essential  to  much  of  what  follows  is  Bayes'  Theorem,  sketched 
here  as  background.  The  conditional  probability  of  an  event  B, 
given  that  another  event  A  has  occurred,  is  symbolized  and  defined 


V  (B  I  (?) 


V0\) 


where  P(A)  (•*£  0)  is  the  marginal  probability  of  event  A,  and 
P(A,B)  is  the  probability  of  joint  occurrence  of  A  and  B.  One  may 
also  speak  of  P(A/B)  =  P(A,B)/P(B)  with  similar  meanings  and 
limits,  leading  to 

P(bia)  PW  «  1.5 

Given  that  B  can  occur  in  n  ways  Bi  ( i=l , 2 , . . . , n )  one  of  which 
always  occurs  with  A,  we  may  sum  expressions  like  Eq.  1.3  for  the 
entire  set  of  events  Bi 

PW2.  in 

os  the  multiplier  of  P(A)  is  equal  to  1,  having  encompassed  all 
possible  pairings.  If  P(A)  ^  0,  we  have  Bayes'  Theorem: 


^Cb-IAV  (A IB l)  "P (B i) 


Suppose  now  that  events  Bi  are  logically  (causally)  prior  to 
event  A.  Then  P(Bi)  is  called  the  prior  distribution  of 
Bi,  P(A/Bi)  the  likelihood  of  A,  given  Bi,  P(A)  the  marginal 
distribution  of  A,  and  P(Bi/A)  the  posterior  distribution  of  Bi . 
Bayes'  Theorem,  given  in  symbols  by  Eq.1.5,  may  then  be  stated  in 
words : 

Posterior  Distribution  =  Prior  Distribution  X  Likelihood  (Function) 

Marginal  Distribution 

(This  argument  holds  for  both  discrete  and  continuous  distributions 
of  probability . ) 

Likelihood  functions  are  a  familiar  staple  of  probability 
theory,  being  forecasts  of  the  frequency  of  chance  events  A  based  on 
presumptions  about  certain  prior  events  or  conditions  (a  die  that  is 
unbiased,  the  "normal"  distribution  of  errors,  half-life  of  a  known 
radioactive  substance).  Marginal  distributions  then  are  forecasts  of 


the  results  of  experiments.  Bayes'  Theorem  tells  us  that  inferences 
about  the  events  Bi  which  lead  to  a  marginal  distribution  cannot  be 
derived  from  the  likelihood  function  alone,  but  require  knowledge  of 
the  prior  distribution  P(Bi)  as  well.  In  the  context  of  our  task,  we 
need  to  know  more  than  the  results  of  a  set  of  missile  firings  to 
infer  the  reliability  of  the  missile. 

Other  requirements  of  a  Bayesian  analysis  will  be  discussed  as 
the  issues  arise. 

Section  3^.  Illustration  of  an  Analysis  in  Accord  with  JCS 
Guidelines 


We  assume  that  the  missiles  and  associated  ground  equipment  used 
in  an  annual  test  do  come  from  a  homogeneous  population,  and  that  the 
several  tests  within  that  year  are  statistically  independent.  We 
assume  further  that  the  reliability  p  is  definable,  and  then  may 
assert  that  were  we  to  know  p,  the  probability  of  s/  successes  and  f(; 
failures  in  n,'  trials  (nl'  =  si'  +  f 1  * )  would  be  by  Bernoulli's 
formula  (a  likelihood  function): 


From  component  testing,  comparison  with  similar  systems, 
comparison  with  other  products  of  the  same  manufacturer,  engineering 
analysis,  we  should  develop  an  estimate  of  p  and  a  measure  of  our 
confidence  in  that  estimate.  Methods  exist,  e.g.  that  of  Maximum 
Entropy  (Reference  10),  for  constructing  from  this  information  a 
function  with  the  properties  of  a  probability  distribution — a  prior 
distribution.  Constraints  of  reasonableness  and  mathematical 
convenience  come  into  the  selection  process.  With  limited 
information  at  hand,  there  may  be  no  unique  solution.  The  analyst  is 
free  to  try  several  priors  and  to  observe  the  sensitivity  of  answers 
to  such  variations. 

Given  a  likelihood  function,  there  can  generally  be  found  a 
"conjugate"  prior  function  (so-called  because  it  marries 
mathematically  to  the  likelihood  function);  properly  a  class  of  such 
functions,  dependent  on  a  limited  number  of  parameters  to  distinguish 
members  of  the  class.  Conjugate  to  the  Bernoulli's  distribution  is 
the  Beta  distribution,  written 

^  1  (s^b)  ~ 
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Different  sets  of  the  parameters  s0  and  f^  give  rise  to  functions 
whose  graphs  are  variously  peaked  at  some  locale  within  the  limits  of 
0  to  1,  are  relatively  flat,  are  J-shaped  and  strongly  peaked  at  0  or 
],  or  are  even  U-shaped  and  strongly  peaked  at  both  0  and  1.  It  is  a 
rich  set  of  functions. 


get 


Taking  the  product  of  dPCs^.f^)  with  the  Bernoulli  function,  we 
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which  when  integrated  over  the  range  of  0  to  1  gives 
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the  marginal  distribution  of  s,’  given  B(s0,f0)  as  prior.  The  ratio 
of  Eqs.  1.6  and  1.7  gives  the  posterior  distribution  of  p  for  s( '  and 
fv  ’  observed  : 
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explaining  my  notation  and  revealing  the  meaning  of  conjugation. 

From  a  prior  distribution  B(s0,f0),  and  a  likelihood  function 
for  a  test  of  a  sample  of  size  n, ' ,  we  have  created  a  function  which, 
as  a  posterior  distribution  from  that  experiment,  is  logically  the 
prior  when  testing  a  second  sample  of  size  n^'.  This  process  can  be 
repeated  ad  libitum,  making  sample  1  refer  to  all  prior  information 
and  sample  2  the  latest  test. 


Now  the  JCS  asks  to  know  the  probability  that  the  reliability  of 
sample  2  (and  by  inference  that  of  the  population  from  which  it  was 
drawn)  is  less  than  a  certain  fraction  k  ( o  <  k  j<  l)of  the 
reliability  estimate  p  of  sample  1.  If  the  evidentiary  basis  for 
this  answer  lies  entirely  in  the  test  of  n2'  items,  then  we  may 
assume  instead  a  uniform  prior  distribution,  drop  the  primes  on  n2', 
s2',  and  f  2 '  and  represent  this  probability  by 

0-  t BU.-U 


which  we  then  integrate  over  the  distribution  of  pi  to  get  the 
probability  that  p2  _<  kpl: 


The  probability  that  p2>kpl  is  just  1  minus  this  result. 


As  an  aid  to  understanding  the  generality  of  this  result, 
consider  the  case  where  pi  =  rl  x  r3  and  p2*r2  x  r3  where  r3  is  a 
reliability  factor  not  subject  to  degradation  but  just  as  much 
subject  to  discovery  as  rl  and  r2.  Within  the  framework  of  Beta- 
function  priors,  we  might  be  led  to  the  posterior  distribution: 

-  K  (t-  rtf' 1  (i-  '  r  ’  (i-  t\)  1  *  0 

where  s3(f3)  is  the  total  number  of  observed  successes  (failures)  of 
the  subsystems  described  by  r3.  For  any  values  of  r3  and  k  between  0 

and  1,  P(p2  <  kpl)  =  P(r2  <  krl).  When  the  latter  function  is  given 

by  integrating  Eq.1.10  first  over  r3  from  0  to  1,  it  is  clear  that 
the  result  is  the  same  as  though  r3  =  1  (i.e.,  it  can  be  ignored). 
Thus  using  the  criterion  p2  kpl  we  cam  be  freed  of  any  concern 
about  reliability  factors  common  to  pi  and  p2.  I  would  assert  that 
this  is  a  good  reason  to  employ  this  criterion  in  preference  to  the 
one  described  next. 

The  JCS  guidance  has  not  always  been  interpreted  as  speaking  to 
a  proportional  reduction  in  reliability;  sometimes  it  has  been 

interpreted  as  measuring  a  reduction  of,  say,  lOOd  percentage  points* 

Instead  of  Eq.  1.9  we  would  then  use 

P(f»-A-)=  f*’~' ('-  p.f'  (*'■,*>■) 
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.  Eqs.  1.9  and  1.11  give  mathematical  meaning  to  the  JCS  guidance. 
If  at  the  chosen  confidence  level  it  is  deemed  that  there  has  been  no 
significant  change  in  the  reliability  between  samples  1  and  2,  then 
sample  2  should  be  merged  with  sample  1  in  preparation  for  the  next 
year's  testing.  Other  criteria  should  be  examined  also  (e.g., 
probability  that  there  has  been  no  significant  departure  from  a 
nominal  value),  but  that  does  not  refute  the  translation  into 
mathematics  of  the  JCS  guidelines. 

At  this  point  I  note  that  much  of  the  historical  course  of 
development  of  mathematics  has  been  devoted  to  a  search  for  solutions 
requiring  a  minimum  of  actual  manipulation  of  numbers.  The 
approximations  used  by  statisticians  are  simply  good  examples  of 
this.  The  ready  availability  today  of  powerful  computers  reduces  the 
need  to  employ  approximations  which  may  be  questionable  in  particular 
cases.  Most  of  the  calculations  to  be  described  here  have  been 
carried  out  on  a  programmable  hand  calculator  (HP-41)  or  home 
computer  (Apple,  Commodore,  etc.).  Accordingly,  the  reader  need  not 
be  concerned  with  an  apparent  intractibility  of  the  formulas.  They 
could  be  evaluated  in  the  field  by  the  troops  of  a  Pershing  fire 
unit. 

There  are  two  matters  of  concern:  the  prior  distribution  and 
limits  to  the  size  of  Sample  1.  I  have  already  discussed  problems 
with  the  prior  distribution.  One  assertion  made  is  that  with 
increase  in  the  size  of  the  data  base  it  can  become  misleadingly 
narrow,  ignoring  "unknown-unknowns."  A  different  way  of  saying  this 
is  that  tests  performed  sufficiently  long  ago  may  be  irrelevant  in 
describing  the  present  state  of  the  missile  inventory;  the  meaning 
of  this  argument  is  that  a  larger  annual  test  size  is  needed  to 
compensate  for  stale  data  in  Sample  1.  The  question  of  test  size 
will  be  the  subject  of  the  following  chapters.  Of  course,  if  there 
is  no  evidence  of  a  change  in  reliability  over  the  years,  there  is  no 
reason  to  purge  old  data. 

Section  Optimum  Test  Size 

In  order  to  determine  the  number  of  missiles  which  must  be 
procured  in  the  next  few  years  to  support  a  test  program  through  a 
long  period  of  service  life,  one  must  have  an  estimate  of  the  average 
annual  consumption  in  testing.  To  get  this  estimate,  especially  if 
it  be  glorified  by  a  phase  like  "optimum  test  size,"  one  must  know 
what  questions  the  tests  are  supposed  to  answer  and  how  frequently. 
This  in  turn  means  "getting  into  the  skull"  of  the  JCS.  We  must 
assume  that  first  of  all  there  is  sufficient  reason  to  conduct  the 
tests,  even  at  the  risk  of  compromise  of  properly-classified 
information.  We  know  that  there  will  be  a  finite  inventory,  and  that 
testing  reduces  that  inventory,  whether  or  not  it  be  formally  divided 
into  tactical  and  non-tactical  portions.  We  can  then  ask  the 


question:  how  does  the  result  of  an  additional  test  change  our 

perception  of  the  system  reliability,  and  so  of  the  sufficiency  of 
the  lesser  inventory  of  missiles  to  conduct  a  military  mission  should 
it  be  committed  to  combat  at  a  future  date?  Possible  answers  are 
discussed  in  Chapter  V.  As  there  are  circumstances  under  which  the 
answer  is  insensitive  to  the  size  of  the  inventory,  we  shall  spend 
more  time  considering  the  case  where  inventory  for  test  has  no 
tactical  mission. 

A  long  string  of  heads  or  tails  when  flipping  pennies  is  not 
impossible  or  even  incredible;  but  after  some  number,  one  is  entitled 
to  wonder  if  the  coin  is  biased.  Similarly,  when  testing  a  missile 
which  is  alleged  to  have  high  reliability,  a  string  of  failures — even 
a  short  one — challenges  the  presumption;  contrariwise,  a  long  string 
of  successes  tends  to  be  uninformative.  In  either  case  there  is  a 
practical  limit  to  the  value  of  the  additional  information  in  an 
outcome  merely  extending  such  a  string. 

To  address  this  problem  we  shall  invoke  the  discipline  of 
Sequential  Analysis,  to  include  Sequential  Probability  Ratio  Tests 
and  test  series  truncation.  Much  of  this  is  "old  hat",  having  been 
developed  in  World  War  II,  most  notably  by  Abraham  Wald  (Reference 
11)  working  on  military  problems,  and  largely  standardized  by  now. 

It  has  recently  been  reported  that  the  methods  were  independently 
developed  simultaneously  by  Alan  Turing  while  working  at  Bletchley 
Hall  to  crack  the  German  ENIGMA  codes  (Reference  12).  More 
importantly  there  is  recent  substantive  new  work  not  yet  "codified" 
in  text  books.  Two  applications  of  sequential  analysis  to  the 
Pershing  missile  test  problem  will  be  presented:  one  by  Nozer 
Singpurwalla  and  Robert  Launer  (Chapter  III)  and  one  by  Michael 
Woodroofe  (Chapter  IV).  While  aspects  of  the  treatment  will  appear 
more  "f requentist"  than  Bayesian,  both  evolve  into  completely 
Bayesian  solutions.  In  this  paper  I  shall  extract  from  their  work, 
and  comment  on  it  as  appropriate.  The  author  of  this  memorandum  is 
not  by  profession  a  statistician,  and  so  requests  that  the  original 
researchers  not  be  blamed  for  errors  in  translating  their  work  into 
this  format. 


Chapter  III 

Launer  and  Singpurwalla '  s  Proposal 


The  following  submission  by  Launer  and  Singpurwalla  is  the 
product  of  over  a  year  of  research  by  the  authors,  initiated  and 
guided  in  discussions  with  the  writer  of  this  note.  I  believe  it 
successfully  addresses  the  problem  placed  before  the  authors.  Note 
that  all  the  appendices  to  this  article  are  to  be  found  at  Appendix 
E. 


As  the  numerical  example  in  the  following  exposition  employs 
fictitious  data  and  arbitrary  values  of  the  parameters  oC  »  ft  »  and  V, 
the  numerical  results  should  not  be  taken  as  applicable  to  the 
Pershing  II  problem.  The  dependencies  and  the  savings  from 
sequential  analysis  are  however  clearly  indicated,  the  penalty  when 
tests  are  batched,  and  the  potential  for  squeezing  information  out  of 
small  samples.  The  next  chapter  reports  further  steps  toward  savings 
through  careful  test  design. 


MONITORING  THE  RELIABILITY  OF  PERSHING  II  MISSILES— 

A  CRITIQUE  OF  THE  CURRENT  METHODOLOGY  AND  A  SUGGESTED 
COMBINED  BAYES IAN-SAMPLE  THEORETIC  APPROACH  + 

by  • 

Robert  Launer* 

Nozer  D.  Singpurwalla** 

1.  INTRODUCTION,  TEST  REQUIREMENTS,  AND  ASSUMPTIONS 

The  reliability  of  the  Pershing  II  missile  arsenal  is  an  unknovn 
parameter  which  presumably  could  change  over  time.  To  monitor  the  re¬ 
liability,  and  also  to  ascertain  the  amount  of  change  in  reliability, 
if  any,  a  sample  of  n  Pershing  II  missiles  is  chosen  from  the  ar¬ 
senal  every  year,  and  each  missile  fired  to  observe  its  success  or  fail¬ 
ure.  The  testing  is  destructive,  and  the  arsenal  inventory  is  not 
replenished.  Thus,  it  is  highly  desirable  to  reduce  the  number  of  test 
missiles  fired  year  after  year.  Also,  if  possible,  it  is  desirable  to 
have  the  total  number  of  missiles  fired' per  year  be  a  multiple  of  three— 
that  is,  3,  6,  9,  etc.  A  stated  requirement  with  respect  to  the  year  by 
year  detection  of  change  in  reliability  is  that  a  change  of  &  should 
be  detected  with  a  probability  of  it  or  more.  Since  the  test  data  are 

+  The  authors'  appendices  are  incorporated  in  this  paper  as  Appendix  E.  DW 
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of  a  pass-fail  nature,  a  correct  probability  model  for  describing  them 
is  the  binomial. 

Our  goal  is  to  determine  a  sample  size  and  a  decision  criterion 
that  will  satisfy  the  above  requirement,  and  minimize  the  total  amount 
of  testing.  Since  each  missile  is  expensive  to  produce  and  test,  there 
is  a  keen  desire  to  incorporate  into  the  analysis  all  knowledge  that  is 

available,  both,  from  the  previous  tests  and  engineering  experience. 

Thus  a  Bayesian  point  of  view  is  natural  here. 

2.  CRITIQUE  OF  PRESENT  METHODOLOGY  % 

Based  on  our  reading  of  the  pertinent  literature  that  has  been 
made  available  to  us,  and  our  discussions  with  several  people  familiar 
with  the  test,  it  is  our  understanding  that  the  current  methodology  for 
analyzing  the  Pershing  II  data  is  based  on  Fisher's  exact  test,  hence¬ 
forth  FET.  We  claim  that  this  technique  is  inappropriate  for  the  situa¬ 
tion  described  above.  Furthermore,  a  modified  version  of  the  FET  which 
has  been  used  in  similar  situations  is  not  appropriate,  either.  Whereas 
the  FET  can  be  used  to  detect  the  equality  or  otherwise  of  two  binomial 
populations,  it  is  not  designed  to  detect  a  specified  difference  between 
the  two  binomial  parameters  in  question.  Furthermore,  FET  does  not  ad¬ 
dress  the  key  question  of  sample  size  selection,  and  thus  fails  to  ans¬ 
wer  the  main  question  posed  by  our  problem.  A  choice  of  the  sample  size 
should  be  based  on  an  assumed  or  target  value  of  the  reliability,  and 
this  is  nowhere  apparent  in  the  test. 

Given  a  sample  size  and  the  test  results  from  this  sample,  the 
FET  can  give  us  the  "p  values"  for  deciding  upon  the  difference  or 
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otherwise  of  the  two  binomial  populations  in  question,  and  this  may  be 
the  sole  motivation  for  using  this  test  here. 

3.  THE  COMBINED  BAYES IAN-SAMPLE  THEORETIC 
APPROACH  PROPOSED  HERE 

Our  proposed  approach  addresses  the  issues  posed  before,  and 
attempts  to  do  this  in  an  economical  manner  with  respect  to  sample  size. 

Since  reliability  changes  over  time,  we  introduce  an  index  t  , 
where  t  =  1,2,...  ;  thus  t  =  1  denotes  the  first  year  of  testing, 
t  *  2  denotes  the  second  year  of  testing,  and  so  on.  Let  nt  denote 
the  number  of  missiles  to  be  tested  in  time  period  t  ;  nt  is  the 
(unknown)  sample  size,  one  of  our  decision  variables.  Let  x^  denote 
the  number  of  missiles  that  fire  successfully  in  time  period  t  ; 
note  that  0  <  xt  <  nt  . 

Let  pt  be  the  chance  that  any  missile  fired  at  t  will  fire 
successfully,  or  its  propensity  to  do  so.  Since  p£  is  unknown  to  us, 
we  express  our  uncertainty  about  it  by  a  probability  distribution,  say 
g(Pt  I  previous  failure  data,  if  any,  and  H)  .  Thus  pt  is  treated  as 
an  unknown  parameter,  and  the  vertical  line  in  g ( • )  denotes  conditioned 
upon  or  given,  and  H  denotes  our  background  information  about  pt  . 

If  we  have  no  previous  failure  data,  then  g(pt  |  H)  denotes  our  prior 
distribution  for  pt  ;  otherwise  g(*  |  •)  denotes  our  posterior 
distribution. 

If  for  each  time  period  t  we  judge  the  missiles  in  the  arsenal 
to  be  exchangeable  (we  have  here  finite  exchangeability),  then  it  is 
appropriate  to  assume  that  given  p£  ,  the  probability  of  observing  x^ 


successful  firings  in  a  sample  of  size 

that  is, 


is  a  binomial  distribution; 
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The  choice  of  the  sample  size  is  based  on  the  following  sample 

theoretic  arguments  for  testing  hypotheses  about  p^  . 

If  p^  ,  the  chance  that  a  missile  is  fired  successfully  at  time 
t  ,  is  large,  then  the  number  of  failures  in  a  sample  of  size  nt  would 
tend  to  be  small.  Given  an  n^  and  having  specified  a  p^  ,  let  x* 
be  the  largest  integer  for  which  the  chance  of  observing  x*  or  fewer 
successes  is  small,  say  a  ;  that  is. 


x*  r 
t  In 


P{x*  or  fewer  successes  in  n  |  p  )  =  \ 

c  c  j=0  j 


t  1  Vj 

pJt  (i  -  Pt)  1  <  a  • 


If  pt  were  to  change  to  p t  -  A  ,  with  A  large,  then  the  num¬ 
ber  of  failures  in  a  sample  of  size  n£  would  tend  to  be  large;  if  A 
were  small,  the  number  of  failures  in  nt  would  tend  to  be  small.  Thus, 
for  some  small  number  6  , 

P{x*  or  fewer  successes  in  nt  firings  |  (pc  -  A)} 


x*  ( 
t  n. 


'I  (P  -  A)J  U  -  P  +  A)  C  >  1  -  6  . 

j-oljj  C 

If  in  (2)  and  (3)  we  assume  that  Pt  »  a  ,  B  ,  and  A  are  the 
only  known  quantities,  then  (2)  and  (3)  can  be  simultaneously  solved  to 
obtain  an  nt  and  x*  .  Once  this  is  done,  (2)  can  be  used  to  test  the 
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is  p  ,  with  a  Type  I  error  a  .  This  is  done  by  accepting  (rejecting) 
the  null  hypothesis  whenever  >(^)  x*  ,  where  x^  is  the  total  number 
of  successfully  fired  missiles  in  a  sample  of  size  nt  •  If  ct  =  .25 
and  B  =  .25,  then  (3)  assures  us  that  nc  and  x*  are  suitable  for 
detecting  the  desired  changes  in  reliability.  Note  that  (3)  describes 
the  power  of  the  test  as  specified  by  (2),  for  changing  values  of  6  . 

If  the  null  hypothesis  is  accepted,  we  conclude  that  the  reliability  of 


the  missile  arsenal  at  time  t  is  Pt  * 

In  our  case  pt  is  not  specified,  as  it  is  an  unknown  parameter 

which  is  liable  to  change  over  time.  What  we  have  instead  is 

\ 

i.  a  prior  distribution  for  at  time  (t-1)  ,  say 

S(Pt  I  *  (n2 »X2^  *  •  •  •  t  (^^2 ^  t  ^  2  and 

g(Px  I  H)  ; 

ii.  a  posterior  distribution  for  pt  at  time  t  ,  say 

g(Pt  I  (n^x  ),  •  ••,  K),  for  t  >  1  . 

Thus,  if  we  uncondition  on  ,  (2)  and  (3)  would  become 

1  x*  . 
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,  for  t  «  1 


In  order  to  obtain  the  pair  (n^.x*)  ,  for  t  ^  1  ,  we  need  to 


solve  (4)  and  (5)  simultaneously.  Note  that  a  solution  to  (4)  and  (5) 


would  depend  on  our  choice  of  g(p^_  |  •)  .  If  for  example,  g(p^  |  *)  is 


a  member  of  the  family  of  beta  density  functions,  then  (4)  and  (5)  would 


involve  incomplete  beta  functions  and  would  call  for  numerical  methods 


for  solving  them.  A  method  for  undertaking  this  is  described  in  Appendix 


A.  A  computer  code  for  implementing  the  method  of  Appendix  A  is  given  in 


Appendix  B.  An  example  using  the  above  is  in  Section  5. 


As  an  alternative  to  the  above,  and  one  which  is  easy  to  imple¬ 


ment,  we  replace  pt  in  (2)  and  (3)  by  pt  ,  the  modal  value  of 
g(pt  I  (n^.x^),  •••»  .  The  modal  value  is  the  most 

likely  value  of  pt  ,  given  all  the  previous  data,  and  is  determined  by 
the  prior  distribution  g(pt  j  (n^.x^),  ...,  W)  .  The 

posterior  distribution  g(pt  |  (n^,x^),  ...,  (nt>xt)>  H )  represents  our 


best  assessment  of  the  arsenal  reliability  at  time  t  ,  given  all  the 


data  up  to  and  including  that  obtained  at  t  .  Its  model  value  pfc 


could  be  used  as  a  single  number  which  describes  p^  .  In  the  next  sec¬ 


tion,  we  discuss  an  implementation  of  the  above  alternative  procedure. 


An  implementation  of  the  main  procedure  follows  along  similar  lines. 


with  the  exception  that  in  computing  the  pair  (nt,x*)  Pt  *s  not 


replaced  by  the  modal  value  of  its  prior  distribution. 


3.1  Assessing  Our  Uncertainty  _ab out  p  _ and  Pr ocedure  Implementation 

Since  can  take  values  between  0  and  1,  a  convenient  but 

flexible  way  for  us  to  express  our  uncertainty  about  p^  is  via  the 
family  of  beta  density  functions  on  (0,1).  Thus, 


1.  We  start  off  our  assessment  and  monitoring  procedure  by 

assigning  a  prior  distribution  for  ,  say  g(p^  |  Y,6,H)  , 

which  for  the  two  unknown  parameters  y  >  0  and  6  >  0  is 


a  beta  density  function 

g(pi  I  y,M)  =  r(Y)r(6)  PI  (1-pi)<S  1  < 

The  modal  value  of  the  above  density  is 


0  <  Pl  <  1  .  (6) 


-  =  Y-l 
P1  y+6-2  ' 

Clearly,  p^  best  describes  in  the  form  of  a  single  number 
our  assessment  of  ,  prior  to  testing  at  time  t  =  1  . 

Furthermore,  p^  is  also  to  be  used  for  determining  the  pair 
n^  and  x*  ,  for  testing  at  time  t  =  1  . 

2.  We  thus  replace  p^  by  p^  in  (2)  and  (3),  and  simultane¬ 
ously  solve  these  to  obtain  and  x*  .  [In  Appendix  A 

we  discuss  how  to  obtain  n^  and  x*  without  using  p^  , 
and  by  directly  solving  (A)  and  (5).] 

3.  We  take  a  sample  of  size  n^  and  test  these  to  determine  x^ 
the  number  of  missiles  that  fire  successfully.  If  x^  >(<)  x* 
we  accept  (reject)  the  hypothesis  that  the  reliability  of  the 


missile  arsenal  at  time  1  is  p  . 

A.  If  we  accept  the  above  hypothesis,  then  we  update  our  opinions 


mmm® 


8 


t 


about  in  light  of  and  via  the  posterior 

distribution  g(p^  |  (n^.x^),  H)  .  The  modal  value  of  this 
posterior  distribution  is 
Y+xl-1 

Pl  =  Y+6+nL-2  ‘ 

and  this  number  best  summarizes  our  assessment  of  after 

testing  at  time  1.  We  now  go  to  step  5. 

5.  If  the  aforementioned  hypothesis  is  rejected,  our  choice  of 

Y  and  6  needs  to  be  revised.  This  should  be  done  follow¬ 
ing  a  more  detailed  analysis  about  p  .  We  then  go  back  to 

«  1 

stage  1. 

6.  The  posterior  distribution  g(p^  |  (n^,x^),  H)  now  serves  as 
the  prior  distribution  for  p2  ,  and  its  modal  value  p^  is 
set  equal  to  p2  .  Thus 

Y+x^l 

P2  Y+S+nj-2  ’ 

and  p^  is  now  replaced  by  p2  in  (2)  and  (3),  which  are 
solved  for  n2  and  x£  .  [In  Appendix  A  we  discuss  how  to 

obtain  n2  and  x£  by  directly  solving  (4)  and  (5).] 

7.  We  now  repeat  the  steps  3  through  6,  and  continue  the  above 
procedure.  Thus,  at  time  ( t— 1)  we  have 

Y  +  +  x2  +  ...  +  xt_j 

Pt-1  Pt  Y  +  6  +  n^  +  n2  +  ...  +  nt_j2  ^ 

as  our  single  best  assessment  of  the  reliability  of  the  arse¬ 
nal  at  time  (t-1)  ,  after  observing  the  results  of  the  test  at 


"■  *  ■  ■  ■  ■  t  ■  r»  ri 


£ 


time  (t-1)  .  It  also  represents  our  choice  for  p  in 
equations  (2)  and  (3),  for  determining  the  sample  size 


and  the  decision  variable  x*  . 


Suppose  that  at  time  t  ,  we  test  n  items,  observe 
successes,  and  based  on  this  result,  reject  the  null  hypothe¬ 


sis  that  p„  =  p„  =  p. 


Then  we  conclude  that  the  reli- 


t  rt  ‘t-1  * 

ability  of  the  arsenal  has  changed  from  its  previous  value 


t-1 


When  this  happens,  we  investigate  the  cause  for  this 


change,  choose  some  new  values,  say  y*  and  6*  ,  and 


estimate  by 


P.  = 


Y’+xt-l 


t  y’+S’+n  -2  * 


We  now  continue  as  before,  bearing  in  mind  that  the  previous 


date  (n^.x^),  are  no  more  appropriate  for 


inclusion  in  our  assessment  process. 

An  alternative  to  the  beta  prior  which  has  properties  of  robustness 
is  currently  under  investigation.  However,  there  is  no  assurance  that 
the  alternative  prior  will  be  void  of  computational  difficulties. 


3. 2  Sequential  Sampling  to  Reduce  the  Amount  of  Testing 

At  any  stage  t  ,  given  an  n^  and  x*  ,  a  further  reduction 
in  the  amount  of  missiles  tested  can  be  achieved  if  the  testing  is  done 
sequentially,  one  item  at  a  time.  Specifically,  we  would  test  one  item 


at  a  time;  and  stop  the  test  as  soon  as  xfc  the  number  of  successes  is 


larger  than  x*  .  Thus,  ideally,  the  number  of  missiles  tested  could  be 


[»] 


as  few  as  x*  +  1  ;  this  implies  a  saving  of  -  x*  -  1  .  The  maxi¬ 

mum  of  missiles  tested  would  of  course  be  no  greater  than  n  .  The 
resulting  sample  size,  that  is  the  number  of  missiles  actually  tested 
at  each  stage  is  known  as  a  curtailed  sample. 

For  the  above  scheme,  given  p  we  can  compute  E (n^_  | P t )  the 
expected  number  of  missiles  tested  using  standard  arguments — these  are 
shown  later.  However,  since  p  is  not  known,  we  average  out  p^ 
with  respect  to  its  prior  distribution  to  obtain  E(n  )  ,  the  un¬ 
conditional  expectation  of  the  number  of  missiles  tested  at  each  stage 
under  the  sequentially  taken  curtailed  sample.  This  is  shown  below. 

Given  n  and  x*  ,  the  probability  that  n  =  x  ,  when  a 
sequential  sampling  scheme  is  used  is 


p{nt=x|pt}  =  < 


x-1 


nt-x*-l 


x-1 


nt-x*-l 


f  x-1  A 


x-x*-l 


Vxt 


(1-Pt)  P 


x-(nf-x*) 


.  n  - 


X*  <  X  <  X* 
t  *=  “  t 


n  -x 


-V* 


(l“Pt) 


(1-Pt) 


t  t  x-(n  -x*) 
P^  t  t 


-x*-l  x*+l 


p  C  ,  x*  <  x  <  n„ 
»  t  —  t 


where 


In  order  to  obtain  P{nt=x)  ,  we  average  out  the  above  by  g(pt|*)  , 


_  ror-nS)  y-im  ,6-1 
e(pt>  5  r(y)r(6)  pt  (1~pc) 
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T(x-n  +x*+y)r(n  -x*+<5) 

t  t  '  t  t 

TCy+S+x) 

for  n  -x*  i  x  1  x* 

r(x-nfc+x*+Y)  (nt-x*+6) 
r(Y+<5+x) 

(x*+i+y) r (x-x*t~l+6) 
r  (y+<s+x) 

for  x*  <  x  n^  , 

from  which  E(n  )  can  be  computed.  The  above  formula  can  also  be  used 
to  plot  a  histogram  of  the  various  values  of  n^  ,  for  each  stage  t  . 

If  the  sequential  tests  are  to  be  done  in  batches  of  3  rather 
than  testing  a  single  item  at  a  time,  the  savings  In  the  number  of  items 
tested  will  be  less.  However,  this  is  still  better  than  compulsarily 
testing  all  the  nt  items.  We  do  not  have  a  general  formula  like  (9) 
above  to  compute  the  expected  sample  size.  The  calculations  will  have 
to  be  done  on  an  enumerative  basis.  These  are  shown  in  Appendix  C. 


When  the  above  is  done,  we  have 

r  r  x-i  \ 


p[nt=x] 


n  -x*— 1 
t  t 


x-1 


nt-x*-l 


r(y-HS) 

r(Y)T(6) 


<-1  'l 


x-x*-l 

t 


Hy+^S) 

tcy) rc6) 


TCy-hS) 

r(Y)T(6) 
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4.  COMMENTS  ON  THE  PROPOSED  APPROACH 

The  proposed  approach  is  a  combination  of  sample  theory  and 
Bayesian  statistics.  The  former  is  used  to  determine  the  sample  size, 
and  the  latter  is  used  for  inference  about  p^  .  One  may  express  reser¬ 
vations  about  a  procedure  in  which  two  philosophical  viewpoints  are  used 
simultaneously.  However,  upon  closer  examination  of  the  approach,  such 
a  concern  should  be  dispelled,  since  the  sample  theory  approach  is  not 
used  for  making  inferences  about  ;  it  is  used  for  choosing  a  sample 

size.  The  selection  of  the  sample  size  after  averaging  out  p  with 
respect  to  its  distribution  g(pt  |  *)  ,  see  equations  (4)  and  (5),  makes 
our  analysis  fall  under  the  category1  of  what  is  known  as  pre-posterior 
analysis,  a  perfectly  legitimate  device  within  the  Bayesian  paradigm 
[c£.  Box  (1982)]. 

The  monitoring  of  pt  is  done  within  the  Bayesian  framework, 
and  besides  "coherence"  it  has  the  advantage  of  inducing  economy  by 
virtue  of  the  fact  that  all  our  relevant  previous  data  are  incorporated 
into  the  analysis.  Furthermore,  it  allows  the  incorporation  of  any 
engineering  or  judgmental  knowledge  that  we  may  have  about  the  missiles 
into  our  analysis  —  this  is  done  via  Che  parameters  y  and  6  or 
y'  and  6’  ,  etc. 

5.  APPLICATIONS  TO  DATA 

Our  proposed  approach  is  designed  to  specify  a  sample  size  for 
testing  at  each  stage,  and  thus  its  effectiveness  cannot  be  fully  ap¬ 
preciated  if  we  apply  it  to  existing  data.  However,  we  shall  apply  it 


Co  some  given  (sanitized)  success  failure  data  to  demonstrate  the  fact 
that  the  computations  of  Appendix  A  can  be  undertaken,  and  to  compare 
the  results  of  our  main  procedure  and  the  simplified  alternative,  de¬ 
scribed  in  Section  3.1.  In  Table  1,  we  present  the  given  success  fail¬ 
ure  data,  our  Bayesian  estimate  of  the  mode  of  p^  at  each  stage  using 
a  uniform  prior  distribution  at  stage  0  updated  at  successive  stages 
using  failure  data,  and  the  values  of  x*  and  N  using  the  main  pro¬ 
cedure  and  the  alternative. 

A  few  facts  emerge  from  an  examination  of  Table  1. 

1.  A  large  number  of  items  to  be  tested  is  called  for,  when 

* 

the  prior  is  uniform,  with  mode  .5  . 

2.  The  number  of  items  to  be  tested  is  the  smallest  when  the 

mode  of  is  closest  to  1,  namely,  at  .9  . 

3.  The  number  of  items  to  be  tested  under  the  main  procedure 
is  always  equal  to  or  larger  than  that  under  the  alternate 
procedure.  This  is  because  the  alternate  procedure  puts  all 
the  probability  mass  at  the  mode,  whereas  the  main  procedure 
disperses  the  probability  mass  over  [0,1]  ,  with  a  concen¬ 
tration  at  the  mode. 

5.1  Results  of  Curtailed  Sequential  Sampling 

The  sequential  sampling  approach  discussed  in  Section  3.2  was 
applied  to  the  data  and  the  results  of  Table  1.  The  nt  and  the  x* 
values  considered  were  those  given  by  the  "alternative  procedure";  this 
procedure  gave  us  smaller  values  of  the  nt's  than  the  main  procedure. 


»* 


TABLE  1 

Results  for  Main  Procedure  and  Alternative,  Using  Sanitized 
Data,  and  Assuming  a  Uniform  Prior  at  Stage  0 


Success  Failure 


Computed  Value 

s  of  X* 

and  n 

t 

Mode 
of  p 

Main  Procedure 

Alt.  Procedure 

t 

X*t 

nt 

nt 

.500 

2 

29 

5 

17 

.875 

8 

13 

9 

13 

.900  , 

10 

1A 

8 

11 

.906 

11 

15 

8 

11 

.909 

8 

11 

8 

11 

.875 

9 

13 

9 

13 

.853 

10 

15 

8 

12 

.825 

9 

14 

9 

1A 

.833 

11 

17 

9 

1A 

.820 

10 

16 

9 

1A 

.837 

10 

15 

9 

1A 

.841 

10 

15 

10 

15 

.836 

10 

15 

9 

1A 

.  848 

10 

15 

8 

12 

.850 

10 

15 

8 

12 

15 
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The  expected  sample  sizes  when  testing  is  sequential,  in  batches 
of  3  as  well  as  one  item  at  a  time,  were  computed.  These  are  shown 
in  Table  2.  The  advantage  of  testing  one  item  at  a  time  is  clear 
from  an  inspection  of  columns  2  and  3  of  Table  2. 

We  also  note  the  overall  reduction  in  sample  si2e  using  the  approach 
of  this  paper.  The  expected  sample  size  can  be  as  small  as  9. 

The  detailed  calculations  leading  us  to  Columns  2  and  3  of 
Table  2  are  given  in  Appendix  C. 


6.  PROPOSED  FUTURE  WORK 

An  objectionable  feature  of>  the  proposed  procedure,  from  a 
Bayesian  point  of  view,  is  the  testing  of  hypotheses  about  p^ 

using  the  decision  variables  x*  ,  t  =  1,2 .  The  proper  Bayesian 

way  to  study  this  problem  would  be  via  a  Kalman  filter  model  which 
contains  two  unknown  states  of  nature,  p  and  m  ,  where  m  denotes 

t  L  t 

the  drift  in  pfc  .  The  Kalman  filter  would  not  only  have  the  ability 
to  monitor  the  reliability  of  the  arsenal,  but  would  also  provide  us 
with  a  vehicle  for  predicting  the  future  arsenal  reliability.  The 
following  are  our  ideas  on  how  a  Kalman  filter  model  for  this  problem 
can  be  developed. 

Let  denote  some  transform  of  xt/nt  »  a*id  one  which  makes 

Y  approximately  normal.  The  observation  equation  for  the  Kalman 


filter  model  would  be 


h  +  ht 


where  *s  a  disturbance  term  with  mean  0  and  variance  . 

We  can  postulate  the  following  as  system  equations: 

Pt  =  mt  +  Y2t  ’  and 

"t  *  "t-l  +  Y3t  ' 


V  V  V  V  VV  V.\-V  VV  V.V  VW'/V  >_V  V“'-'  WV^,-  V.VIW^V 
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TABLE  2 

Expected  Sample  Size  for  Curtailed  Sequential  Sampling  in  Batches  of 

Size  3  and  Size  1. 


Stage 

t 

Expected  Sample  Size 
for  Batch  Size  3 

i 

ssssssaa  ■.  ssaaaBssggssssa  sss-sarsrsssg-s; 

Expected  Sample  Size 
for  Batch  Size  1 

| 

1 

X* 

c 

nt 

0 

11.84 

10.91 

5 

17 

1 

12.03 

10.66 

| 

9 

13 

1 

2 

10.29 

9.45 

8 

11 

3 

10.37 

9.51 

8 

11 

4 

10.40 

,  9.54 

8 

11 

5 

12.28 

11.08 

9 

13 

6 

11.07 

10.16 

8 

12 

7 

12.84 

11.74 

9 

14 

8 

12.79 

11.69 

9 

14 

9 

12.87 

11.78 

1 

9 

14 

10 

12.78 

l 

11.67 

9 

14 

11 

13.59 

12.72 

10 

15 

12 

12.78 

11.68 

9 

14 

13 

11.14 

10.22 

» 

8 

12 

14 

11.14 

10.21 

8 

12 

♦ 

I 

k 


• 

i 

A  I 


1 


if; 


$ 

£ 

f 
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In  che  above  equations,  we  are  saying  that  p  ,  the  unknown  state  of 
nature,  consists  of  a  low  frequency  drift  term  ,  which  represents 

a  smooth  variation  in  p^  ,  and  *  which  is  a  high  frequency  compo¬ 
nent  that  represents  drastic  changes  in  pt  .  We  assume  that  is 

2 

a  normal  variate  with  mean  0  and  variance  Oj  .  The  drift  term  is 

assumed  constant,  except  for  slight  disturbances  in  it;  these  are 

described  by  y^  ,  which  is  also  assumed  normal  with  mean  0  and  vari- 
2 

ance  - 

The  Kalman  filter  solution  would  result  in  uncertainty  statements 
about  pt  and  m  ,  via  their  distribution  functions.  These,  of  course, 
would  be  conditioned  on  (n^Xj),  .  ..*,  (nt,xt>  .  Large  values  of  mt 

would  indicate  a  drift  in  the  arsenal  reliability,  and  so  m^  could  be 
used  to  monitor  the  change  in  the  arsenal  reliability. 

It  appears  that  the  Kalman  filter  solution  would  have  several 
advantages  over  the  proposed  approach.  The  problem  of  choosing  n£  in 
the  context  of  a  Kalman  filter  is  an  open  question,  and  this  calls  for 
some  basic  research,  assuming  that  this  has  not  been  done  before. 

A  third  possible  direction  for  future  research  is  the  development 
of  a  sequential  procedure  for  testing  the  missiles.  A  sequential  proce¬ 
dure  employing  Bayesian  considerations  may  add  a  further  dimension  to 


$ 

i 


& 


this  problem. 


Chapter  IV 

Woodroofe’s  Proposal 


The  proposals  of  Michael  Woodroofe  are  not  yet  formally 
documented,  but  are  contained  in  a  series  of  letters  and  lecture 
notes  (References  13-17).  In  this  chapter  I  shall  mostly  quote  from 
this  material  with  the  author's  permission,  noting  that  any  published 
versions  may  differ  markedly  from  those  given  here.  I  accept 
responsibility,  however,  for  the  accuracy  of  the  material  quoted  and 
the  interpretations  and  extensions  of  it. 

All  of  the  calculations  described  in  this  chapter  were  carried 
out  by  Dr.  Woodroofe  and/or  myself.  I  have  programmed  most  of  them 
for  an  HP-41C,  and  listings  are  given  in  Appendix  D.  Instructions 
and  copies  on  magnetic  cards  are  available.  Dr.  Woodroofe  has  used 
an  Apple  computer. 


Section  1 .  (Extract  from  Reference  14). 

The  Truncated  Sequential  Probability  Ratio  Test. 

Illustration  with  a  sequential  test  of  the  type  of  savings 
which  are  possible  and  the  loss  of  information  which  results  from  the 
savings.  Note  that  the  process  starts  with  the  conventional 
Uniformly  Most  Powerful  test,  to  be  terminated  when  a  specific  number 
Sn  of  failures  has  been  observed;  or  when,  out  of  a  planned  test  of 
size  n,  the  number  of  observed  successes  assures  that  the  number  of 
failures  cannot  reach  Sn ;  or  after  n  tests  if  not  terminated  earlier. 
The  choice  of  n  is  at  this  time  arbitrary;  the  value  12  was  used  in 
the  example  to  permit  comparison  to  the  Pershing  test  program,  past 
and  planned. 


/We  start  with  a  discussion  of/  the  problem  of  sequentially  testing 
/such- that/  that  a  failure  probability  does  not  exceed  a  given  level.  I  wi] 
illustrate  the  type  of  savings  which  are  possible  and  the  loss  of  infor¬ 
mation  which  result  from  the  savings  with  a  specific  example. 


Let  X.  ....  X.,  be  ! .  1  .d/random  variables  which  take  the  values 


9  •••  a.  M  i  •  i«u.  i  aiiuuui  var  iodic)  wnicn  iokc  me  value 

I  and  0  witn  probabilities  p  and  q  *  1-p  ,  where  0  <  p  <  -1  ,  Is 
unknown;  and  consider  the  problem  of  testing 


H  :  P  <  .15  . 


Let  $k  -  Xj  ♦  ...  ♦  Xk  , 


1  <  k  <  12  . 


Then  the  (UMP)  .test  which  rejects  H  If  and  only  If  S, k  has  power 
function  °  ,z  “ 


*°,p)  ■ 1  -  io  CO pk  ,n'k  •  ° 


<  p  <  i  . 


Of  course,  it  may  not  be  necessary  to  take  all  12  observations  to  determii 
whether  Sj2  >_  M  .  The  test  may  be  curtailed  at  time 


mln{k  >_  1 :  S.  A  or  S.  <_  k-9}  . 


(2)  E(t  )  -  l 
p  °  k-4 


£  k(kj') 


k  k-1* 

p  q 


l  <v) 


n9  nk"9 

q  p 


o  <  p  <  1 


*  Identically  and  Independently  Distributed. 
**  Uniformly  Most  Powerful. 


1 


r 


Is  the  expected  sample  size  of  the  curtailed  test. 


Selected  values  of  3  (p)  and  E  (t  )  are  listed  In  columns  2 

o  p  o 

and  4  of  Table  1  below. 

Observe  that  the  type  I  error  probability  Is  .0922  when  p  *  .15  and 
the  type  II  error  probability  is  .2253  when  p  ■  .*». 

I  tried  to  construct  a  truncated  version  of  the  SPRT  whose  power 
function  matched  3q  as  closely  as  possible.  Wald's  approximations 

allow  one  to  match  the  power  function  at  two  points.  I  picked  .15  and 
.J|0.  Wald's  approximations  then  give  formulas  for  the  upper  and  lower 
stopping  boundaries  in  the  (k,  S.)  plane.  These  are  listed  In  columns 
2  and  3  of  Table  2.  There  are  two  problems  with  these  boundaries:  Wald' 
approximations  tend  to  overestimate  the  error  probabi 1  it les;  and  1  wanted 
the  test  to  take  at  most  12  observations.  After  some  experimentation 
with  formulas  (3)  and  (A)  below,  I  was  led  to  the  upper  and  lower 
boundaries  listed  In  columns  4  and  5  of  Table  2. 

Thus,  I  considered  the  sequential  test  which  takes 

t  -  min{k  ^  1 :  or  Sk  b^} 


observations  and  rejects  H  If  and  only  If  S  >  b  ,  where  a.  and 
b,  are  as  In  Table  2. 


Let 


The  power  function  and  expected  sample  size  may  be  easily  computed. 


fk(J.p)  -  .  t  >  k) 

r  *  . 


for  k  »  0  ,  ...  ,  11  ,  J  •  0,  1,2,...  ,  and  0  <  p  ■<  1  .  Then  the 
power  function  and  expected  sample  size  are 

11 


(?) 

and 

($ 


B(P>  “  2  P>  *  P  ” 

k-1  K  '  K 


12 


Ep(t)  “  kl}  k{fk-l(bk'1’  P)  P  +  fk-l(V  p)q} 


for  0  <  p  <  1  .  Thus,  one  need  only  compute  the  values  of  f^  ;  and  thi! 
is  easy  in  view  of  the  initial  conditions,  f  (0,p)  ■  1  and 

#  f  I  \  f _  *  /  A  _ _ _ • _  0 


fQ(J,p)  ■  0  for  j  f  0  ,  and  the  recurs  i 


on 


(5)  fk(J,p)  “  fk-l^"l,p*  +  q  fk-l*J,p^  ‘*ak  <  J  <  bk^ 


'v. 


I 

\L\ 

vS 

a 

II 

1 

$ 


S'J 


i.o 

ii 


v  \for  k  -  1  .....  12  ,  J  -  0,  l,  2, 
denotes  the  Indicator  of  A. 


,  and  0  <  p  <  1  .  Here  I 


The  power  function  and  expected  sample  size  may  be  computed  from 
(3).  (**),  and  (5).  Selected  values  are  listed  In  columns  3  and  5  of 
Table  1. 

Observe  that  the  power  functions  3  and  3  differ  by  at  most 
.0103  for  the  values  computed.  This  Is  °  much  better  than  I  had 
expected  when  I  began  the  exercise.  Observe  also  that  the  expected  I 
sample  size  of  the  modified  SPRT  is  substantially  smaller  than  that  of  j 
the  curtailed  test  when  p  Is  small. 

After  the  test  has  been  performed,  one  may  set  confidence  limits 
for  p  by  using  the  relationship  between  tests  and  confidence  intervals. 
Order  the  possible  outcomes  in  a  clockwise  manner,  as  in  column  1 
of  Table  3-  For  each  r  ,  0  <  r  <  1  ,  one  may  test  the  hypothesis 

Kr:  Plr 

as  follows:  the  acceptance  region  A(r)  of  the  test  consists  of  an 
Initial  segment  of  outcomes,  In  the  order  of  Table  3;  one  Includes 
precisely  enough  outcomes  to  make 

Pr(A(r)  )  >  .90  . 

Then,  after  the  test  has  been  performed,  an  upper  confidence  bound  p 
for  p  may  be  obtained  from  the  relation 

p  <  p*  Iff  (t,St)  c  A(p)  . 

This  Is  essentially  the  approach  of  Slegmund  (1978,  Blometrlka),  but 
substitutes  exact  calculations  for  his  approximations. 

I  list  some  approximate  75%  upper  confidence  bounds  for  p  In  Table 
These  were  obtained  by  linear  interpolation  with  formulas  like  (3). 

To  the  extent  that  the  modified  sequential  test  takes  fewer  observa¬ 
tions  than  the  curtailed  test,  one  may  expect  less  accurate  estimation  of 
P- 


Here:  Column  1  Is  computed  from  (l),  column  2  from  (3),  column  3 
from  (2),  and  column  4  from  k. 


Table 

2:  Upper  and  Lower  Stopping 

Boundaries  In  the 

(K.  Skl 

PI  an« 

The  SPRT 

Modified 

k 

* 

\ 

* 

ak 

bk 

1 

-1 

2 

3 

2 

-1 

3 

-1 

3 

3 

-1 

3 

“1  * 

3 

k 

-1 

3 

-l 

<t 

5 

0 

3 

-1 

It 

6 

0 

it 

0 

* 

7 

0 

k 

0 

k 

8 

1 

k 

0 

k 

3 

1 

it 

l 

it 

10 

1 

5 

l 

<t 

ll 

1 

5 

2 

U 

12 

2 

5 

3 

<t 

Here  columns  2  and  3  are 
are  ad  hoc  approx Imat Ions 

from  Wald' 

• 

s  approximations; 

columns  i| 

and  .5 

n 


Outcome 


Confidence  Bound 


Comment,  by  L/W : 


As  indicated  in  Chapter  III,  expectations  of  *  and  E  can  be 
computed  based  on  a  prior  probability  distribution.  Closed-form 
solutions  exist  for  o  and  Ep(to)  for  a  Beta  prior,  among  others, 
lor  '■  ( p )  ,  and  Ep(t),  numerical  integration  is  necessary.  Other 
indices  derived  from  the  f k ( j , p )  in  manners  like  that  for  v  or  E(t) 
can  also  be  meaningfully  be  averaged  over  a  prior  distribution.  As 
Ep(t)  has  here  a  narrow  range  of  variation,  its  expectation  value 
i  1 1  not  be  very  sensitive  to  the  choice  of  the  prior  distribution. 


1)  Testing  Hq:  0  >  .15  is  the  same  as  testing  6'  =  1  -0  <  .85.  If 
you  want  to  have 

Pgtdecide  0'  >  .85)  <  qq  for  0'  <..85 


P0 {decide  0'  <  .85)  <  for  all  0*  >  >  .85, 

where  cxq  and  are  small  and  .85  <  0^  <  1,  then  you  cannot  simply  reverse  the 

roles  of  zero  and  1  in  the  test  described  in  my  earlier  letter.  A  new  test 
must  be  constructed.  See  (2)  below. 

*n  Z^ect^on  ®  was  the  probability  of  a  system  failure. 

2)  For  testing  Hq:  0  <  0()  at  level  oo  with  type  II  error  at  most  aj 
when  0  >  0j,  where  0  <  0q  <  @i  <  1  are  specified,  the  SPOT  continues  sanpling 
as  long  as 


1/A  <  Ln  <  B 


where  B  «=  (l-a2.V<*0»  A  =  (l~aoV^l»  Ln  is  the  likelihood  ratio.  One  finds 


in  =  exp  U]Sn  -  n  Lq) 


where 


.  Aj  =  log  eid-flo*  ~  log  6o(l_0l) 


Aq  =  log  (1-8q)  -  log  (l-ej.) 


and  Sn  =  n  >  1. 


Since  Sn  are  integer  valued,  equation  (*)  may  be  rewritten 

an  ^  ^  ^ 

an  "  1  ij(n*0  -  109  A,) 

*  di  u 

bR  *  i  j  (nAQ  +  log  B)1  +  1 

where  tx]  is  the  greatest  integer  which  is  less  than  or’  equal  to  x. 

Suppose  now  that  one  wants  the  test  to  be  truncated  at  M  say.  Then  one 
wants  boundaries  an  and  1  <  n  <  M.  What  I  did  in  the  example  was  the 
following.  Let  and  be  such  that 

aM<aM  =  bMr’landbh“^' 

say  two  integers  near  the  middle  of  the  interval  frcra  aj^  to  Then  let 

a„  =  max  (a  ,  aj-  CM  -  n)  \ 
n  n  w 

and  b  =  min  {b  ,  bM) 

n  n  n 


for  n  <  M.  This  gives  a  first  approximation  to  the  boundary.  In  the  exampl 
I  then  corputed  the  power  function  of  the  sequential  test  with  boundaries  ar 
and  bjj  and  compared  it  with  the  power  function  of  the  fixed  sample  size  test 
I  then  changed  a  few  of  the  boundary  points  to  get  better  agreement  between 
the  two  power  functions.  The  adjustments  were  minor  and  tended  to  make  the 
continuation  region  fatter. 

i 

The  reason  that  you  can't  pin  me  down  on  the  adjustments  is  that  it  is 
trial  and  error  operation. 


(3)  In  the  example, 


Pe{t=k,Sk  =  bfc}  -  ffc-ifbjc  ~  1; 6) •  B 


and 


Pe{t=k,Sk  =  ak)  =  fk-l^akJ®)*  (l-0) 


Then  Pe{Xt  >  x}  is  the  sum  of  these  probabilities  over  all  pairs  (k,aj.)  and 
(k,^)  for  which  a^/k  >  x  or  b^/k  >  x. 


4)  For  inverse  sampling  there  is  just  one  boundary.  Fbr  curtailed 
sampling,  there  are  two.  Let 

t+  =  min  Ik  >1:  Sk  >  4} 


and 


t“  ^  min{k  >1:  k  -  Sk  >  9} 


Then  Ee(t+)  =  4/B 

and  Ee(f)  =  9/(l-8) 

The  stopping  time  for  the  curtailed  fixed  sample  size  test  is 

to  =  min(t+,t*") 

I 

So  Ee(to)  <  min{E0(t+)^»Ee(t~)} 

When  6  =  .15,  Ee(t~)  *  10.6. 

The  formulas  for  E@(t+)  and  Eg(t”)  hold  for  all  6|  0  <  6  <  1. 

5)  I  think  of  the  boundaries  as  a  modified  S.P.R.T.  In  the 
example,  they  were  similar  to  the  curtailed  fixed  sample  size  test,  but 
sufficiently  different  to  reduce  the  expected  sample  size  by  about  1  over  the 
range  of  interest. 

e *)  The  calculations  in  my  letter  to  Launer  are  for  fixed  6..  To  do 
a  Bayesian  calculation,  one  would  average  them  over  0  values 


The  formulas  which  I  gave  for  computing  the  power  and  expected  sample 
implicitly  assume  that  that  the  boundaries  an  and  bn  are  non-decreasing  in  n. 


Section  3  (Extract  from  Reference  16). 

The  Truncated  SPRT,  Aggregated  over  Several  Tests. 

Derivation  of  a  conservative  estimate  of  the  probability 
that  in  10  years  of  testing,  at  12  missiles  planned  for  expenditure 
each  year,  no  more  than,  say  100,  will  be  needed  using  the  proposed 
stopping  rules. 


This  is  to  explain  haw  savings  in  expected  sanple  size  may  be 
translated  into  savings  of  units  which  must  be  purchased  prior  to  the  . 
experimentation.  Fbr  definiteness,  I  illustrate  the  method  with  the  — 
truncated  SPOT,  which  is  described  in  /  Section  IT  _ ” 

In  particular,  recall  the  computation  of 

f(k,j;p)  *  PR(T>k,Sk=j), 

where  p  denotes  the  true  failure  probability,  denotes  the  number  of 
failures  after  k  units  have  been  tested,  and  t  denotes  the  stopping  time. 
From  this,  one  gets  V 

'  k 

G(k;p)  «=  Pr(T<k)  -  1  -  Ij=o  fOtr  j?p) 
and  g(k;p)  =  Pr(T=k)  =  G(k;p)  -  G(k-l;p) 
for  k  *  1,...,12  and  0  <  p  <  1. 

Suppose  that  the  truncated  SPOT  is  run  n  times,  say  onoe  each  year 
for  n  years,  where  n  is  a  positive  integer.  Then  there  will  be  a  sequence 
Pl»***»Ph  unobservable  true  failure  probabilities  and  a  sequence  tj,..., 
of  random  sample  sizes.-  Here  I  regard  P^f-rPn  as  unknown  parameters,  and 
suppose  that  ti*..*#^  are  independent  random  variables  for  which 

Pr(tj=k)  =  g(k;Pi) 

for  k  =  1,...  ,12  and  i  ■  l,...,n.  If  Pir**-/Pn  are  really  random  variables 
then  the  calculations  described  below  are  valid,  if  the  conditional  distri¬ 
bution  of  ti,...,^  given  Pi».**,PVi  is  3ust  described. 


2 


10 


Let  T  denote  t)ic  total  number  of  units  used  during  the  tests, 

T  =  tj+. .  •+tn. 

Then  the  distribution  of  T  is  required.  The  distribution  of  T  is  the 
involution  of  the  individual  distributions  of  tj,...,^.  This  depends 
on  pi .  ,Pn  in  a  complicated  manner,  but  it  is  possible  to  find  the  sharp 
bound  which  is  valid  for  all  possible  choices  of  Pi,...,Pn.  That  is,  it  is 
possible  to  find  a  function  H  for  which 

Pr(T<k)  >  H(k) 

for  all  k  =  l,...,12n  arv3  all  possible  choices  of  Pi,...,Pn. 
r-  I  describe  the  derivation  be  lew, 

1  vaiues  0f  h]  are  included  in  Table  2  in  the  special  case  that 

" — n  =  10.  Observe  that  then 

Pr(T  >  105)  <  .054 

for  all  PI  .  ,Pn«  The  bound  is  reasonably  sharp,  since  Pr(T>105)  =  .050 
when  all  Sf'pi . Pn  “f3*1  •27‘ 

While  the  bound  is  sharp,  the  approach  is  conservative,  since  it 
iqnores  data  from  previous  years  and  assumes  the  worst  possible  values  for 
p, , . . .  ,p_.  If  an  independent  verification  is  required  for  each  year,  then 

sane*  of  this  conservatism  may  be  unavoidable. 

The  derivation  of  the  bound  uses  the  notion  of  stochastic  dominance. 

If  X  and  Y  are  random  variables  with  distribution  functions  F  and  G,  then  Y 
is  said  to  be  stochastically  larger  than  X  if  and  only  if  G(z)  <  F(z)  for 
all  z.  If  X  and  X'  are  independent  random  variables  and  Y  and  Y*  are 
independent  randan  variables  and  if  Y  and  Y*  are  individually  stochastically 
larger1^®  X  and  X',  then  Y+Y*  is  stochastically  larger  than  X+X'  (as  is 
easily  verified);  and  this  result  extends  from  two  summands  to  several. 

To  apply  this  result,  let 

G(k)  =  min  G(k;p)r 

where  the  minimum  extends  over  0  <  p  <  1.  Then,  for  any  choice  of  Pi, ...,Pn, 
the  distribution  of  T  is  stochastically  dominated  by  the  sum  of  n  independent 
random  variables  having  common  distribution  function  G.  Computing  G  is 
straightforward.  For  k  <  6,  the  minimum  is  attained  when  p  =  0  and  G(k)  «  0. 
For  k  >  6,  I  computed  G(k;p)  for  a  grid  of  p  values  and  took  the  minimum  over 
this  grid.  The  values  nre  listed  in  Table  1.  I  used  a  grid  width  of  .01. 


s&B5s 


re  ( 

.V.. 


Minimum 

Mean  and  St  dev 


TABLE  1.  Values  of  G(k;p) 


6 

7 

8 

9 

10 

11 

2313 

.2590 

.2966 

.5032 

.5556 

.7723 

2222 

.2535 

.2955 

.4967 

i  5537 

.7685 

2144 

.2496 

.2962 

.4923 

.5538 

.7661 

2081 

.2474 

.2987 

.4900 

.5559 

17651 

2032 

♦_2468 

.3030 

14897 

.5599 

.7654 

1996 

.2477 

.3088 

.4914 

.5657 

.7669 

1974 

.2501 

.3163 

.4949 

.5731 

.7696 

1964 

.2540 

.3252 

.5002 

.5819 

.7735 

,1967 

.2593 

.3355 

.5072 

.5922 

.7783 

1981 

.2659 

.3472 

.5156 

.6036 

.7840 

Oc 

,1964 

b 

.2468 

.2955 

A 

.4897 

Q. 

.5537 

X 

.7651 

v  =  9.4528  ~  o  =  2.1992 


—  1*2 *  \  —  (^b*  t«r  A*  ci 

=  )"2.  "  +  <t-  Oo+q)<k-  C<)-n-)<L- b—  C>*t^ 

Notes:  G(12;p)  ■  1  for  all  0  <  p  <  1;  the  minimum  is  zero  for  k  <  5;  y  and  o 
are  the  standard  deviation  of  the  minimizing  distribution. 


0 


TABLE  2.  Values  of  H 


( 


1  -  H(k) 


100 

.2026 

101 

.1622 

102 

.1273 

103 

.0978 

104 

.0734 

105 

.0537 

106 

.0382 

107 

.0263 

108 

.0175 

109 

.0112 

110 

.0069 

111 

.0040 

112 

.0022 

113 

.0012 

114 

.0006 

115 

.0002 

H(k)  -  H(k-l) 

.0460 

.0404 

.0349 

.0295 

.0244 

.0197 

.0155 

.0118 

.0088 

.0063 

.0043 

.0029 

.0018 

.0011 

.0006 

.0003 


Comments  by  DW: 

Let  g ( k )  =  G(k)-G(k-1) . 


4.1 


Then  d(n,z)  =  z^g(n-k) 

4.2 

f 

Jc=o 

is  a  generating  function  of  the  distribution  g(k). 
function  for  the  dominant  of  m  years’  test  results 

The  generating 
is  then 

>ry'\  -  ~ 

rim 

J=o 

^  z  > 

4.3 

and  the  dominant  of  the  probability  that  a  specific  number  J  of  tests 
can  be  forgone  is  given  by  the  coefficient  dJ  of  zJ  in  the  expansion 
of  D(n,m) . 

In  our  example  n  =  12,  and  the  g(k)  for  k  <  6  are  all 
zeros.  Sample  data  are  given  in  Table  3.  So,  for  m=10, 


D  = 

h,0  j.)  -  z  ^0  0  21  <5  Go) 


4.4 


-v  ^ 


* 


\o 


lO?r 


<3  (9)  +■  ( »+zr5(f ) 


lo 


TABLE  3 


g(k) 


P  =  .85 

P  =  .75 

Batch  Size 

Batch  Size 

1  3 

1  3 

.2349 

.5103 

.0940 

.4433 

.2114 

0 

.1258 

0 

.0640 

0 

.2235 

0 

.1942 

.2933 

.1694 

.3604 

.0487 

0 

.0361 

0 

.0504 

0 

.1549 

0 

.1964 

.  1964 

.  1963 

.1963 

In  U'oodroofe  s  notation 


In  particular,  in  our  case, 


4  =  U6z.o)-  vA0*<0 1  l-  hO,c))=[V'i)1 


is  the  dominant  of  the  probability  that  all  120  are  required  (none 
can  be  foregone).  It  follows  that 

23s  U  (wn-|-T) 


ls  the  dominant  of  the  probability  that  at  most  J  can  be  forgone;  the 
generatingfunctionforeJis 


The  calculation  of  the  dJ  or  eJ  presents  no  difficulty  except 
possibly  in  the  control  of  round-off  errors  for  J  large.  Sample 
results  are  given  in  Tables  4  and  5  partly  repeating  material  in 
Table  2,  with  differences  presumably  due  to  differences  in  accuracy 
between  our  computers. 


In  actual  conduct  of  Follow-on  Tests,  three  failures  in  a  row, 
or  two  with  an  identifiable  cause,  would  be  sufficient  justification 
for  halting  the  test  until  the  problem  were  (identified  and)  fixed. 
There  would  then  remain  some  number  of  missiles  from  that  year's 
allocation  available  for  intensive  investigation  of  the  fault  and  for 
demonstration  of  remediation.  It  is  not  clear  that  any  additional 
missiles  would  need  to  be  allocated  to  those  missions,  as  they  could 
serve  the  FOT  mission  at  the  same  time. 


It  is  a  trivial  matter  to  revise  the  expression  for  D(n,m)  to 
treat  the  case  of  batched  tests:  for  example,  in  groups  of  3. 

Tables  3-5  compare  the  results  for  single  and  triple  tests.  For  the 
data  in  the  example,  whatever  the  number  of  missiles  considered  an 
adequate  inventory  for  10  years*  testing  without  batching,  about  6-10 
more  would  be  required  when  fired  in  batches  of  3.  The  analysis  in 
Chapter  III  gave  a  similar  result. 


Up  to  this  point  the  development  has  assumed  that  up  to  12 
would,  in  fact,  be  expanded  if  necessary  to  provide  the  foundation 
for  an  annual  confidence  estimate.  The  question  now  is:  why 


Singles 


TABLE  4 


P  =  .85 


Batches  of  3 


k  dJ= 

e  J  = 

d  j 

H(k)-H(k-1 ) 

l-H(k) 

120  5.1E-7 

5.1E-7 

.0012 

119  4.5E-6 

5 . 1 E-6 

118  2.0E-5 

2.5E-5 

117  .0001 

.  0001 

.0069 

116  .0001 

.0002 

115  .0003 

.0006 

114  .0006 

.0012 

.0224 

113  .0011 

.0022 

112  .0018 

.0040 

111  .0029 

.0069 

.0511 

110  .0043 

.0112 

109  .0063 

.0175 

108  .0088 

.0263 

.0902 

107  .0118 

.0382 

106  .0155 

.0536 

105  .0196 

.0733 

.1291 

104  .0243 

.0975 

103  .0292 

.  1268 

102  .0342 

.1609 

.  1545 

101  .0392 

.2001 

100  .0439 

.2441 

0012 


.0081 


.0305 


.0816 


.1718 


.3010 


.4554 


TABLE  5 
P  =  .75 


Singles 

Batches  of  3 

d  J= 

e  J= 

dJ 

eJ 

H(k)-H(k-1) 

l-H(k) 

5E-11 

7E-10 

6E-9 

.0003 

.0003 

3E-8 

.0024 

.0027 

1 . 4E- 7 

5. 5E-7 

2.0E-6 

5.0E-6 

1 .4E-5 

0 

.0100 

.0127 

3.0E-5 

.0001 

.0284 

.0411 

S  jE 

5  3? 


annually?  If  an  annual  series  should  end  without  clear  resolution, 
as  indeed  it  must  occasionally  according  to  the  current  plans  what 
then?  If  there  is  not  a  clear  cause  of  alarm,  there  is  no  need  for 
alarm . 


Consider  a  decision  to  limit  the  annual  expenditure  to  9 
missiles,  while  extending  the  reporting  period  to  cover  12  missiles 
(the  current  standard)  if  uncertainty  had  not  been  earlier  resolved. 
In  the  worst  case  (all  12-missile  series)  reports  would  occur  at  16- 
month  intervals,  or  8  reports  in  11  years.  Were  the  JCS  to  accept 
biennial  reporting  as  an  (occasional)  substitute  for  annual 
reporting,  this  would  be  a  technically  simple  solution. 


A  Completely  Bayesian  Stopping  Algorithm 


[This  is  my  suggestion  for  doing  a  complete  Bayesian]  decision  theoretic 
analysis  of  the  stopping  problem.  On  the  basis  of  the  preliminary  calculations 
described  below,  I  estimate  that  this  approach  would  reduce  the  number  of  units 
needed  for  testing  by  at  least  one  per  year  over  the  savings  which  may  be 
attained  by  using  a  sequential  probability  ratio  test. 

The  approach  requires  the  specification  of  a  prior  distribution  and  a 
loss  structure.  I  suggest  a  possible  form  for  these  quantities  below;  but 
other  choices  would  yield  to  similar  analyses. 


Let  p  denote  the  proportion  of  non-defective  items  in  the  population. 

Let  hj  denote  a  density  on  the  unit  interval,  0<p  <1;  let  hg  denote  the 
uniform  density  on  the  unit  interval;  and  consider  prior  densities  of  the  form 

(1)  g(p)  =  w  h1(p)  +  (l-w)h0(p), 

where  0  <  w  < 1  is  a  prior  parameter.  Here  may  be  thought  of  as  the 
posterior  density  which  resulted  from  last  year's  tests,  and  w  is  the 
probability  that  p  hasn't  changed  during  the  past  year.  If  p  has  changed, 
which  it  may  with  probability  1-w,  then  it  is  assumed  to  be  uniformly 
distributed  over  the  interval  0  <  p  <1. 


Suppose  now  that  one  may  observe  conditionally  independent  Bernoulli 
randon  variables  X^,...,X^  with  common  success  probability  p,  given  p,  and  let 


sk  =  xi+...+Xk 

denote  the  number  of  successes.  Then  the  posterior  distribution  of  p,  given 

X,,...,X  is 
1  n 


gk<P)  =  w  h^  (p)  +  (l-w)hg(p) 


k-S, 


where  h^(p)  =  h^Cpjk.S^)  a  p  ^(1-p)  Ni^(p) 
l  h^(p)dp=l 


and 


^  Suppose  now  that  a  critical  level  Pq  is  given  with  the  following 
properties:  if  p  >  Pg,  then  the  population  contains  enough  good  items;  if 
p  <  pQ,  then  the  population  no  longer  contains  enough  good  items  and 
corrective  action  is  desirable;  and  if  p  is  nuch  less  than  pQ,  then  corrective 
action  is  necessary.  Suppose  further  that  the  purpose  cf  each  year's  test  is 
to  decide  whether  P  <  Po  or  p  >.pg;  and  define  one  unit  of  cost  to  be  the  cost 
of  testing  one  iten.  Then  the  decision  problem  may  be  modelled  as  follows: 
the  possible  decisions  are  1  to  decided  that  p  <  PQ  and  2  to  decide  that 
p  >  pg;  if  one  decides  that  p  <  pg  when,  in  fact,  p  >pg,  then  one  loses  Cp 
units;  and  if  one  decides  that  p  >  pg,  when,  in  fact,  p  <  pg,  then  one  loses 
C2(po~P)  units.  Hare  Cp  and  C2  are  positive  constants.  Cp  represents  the 
cost  of  inspecting  the  entire  system;  and  the  ratio  C2/Cp  is  determined  by  the 
relative  importance  of  the  two  kinds  of  errors. 

These  three  elements,  the  prior  distribution,  the  sampling  distributions, 
and  the  loss  structure,  determine  an  optimal  sampling  plan,  one  vhich 
minimizes  the  sum  of  sampling  oosts  and  expected  loss  to  due  an  incorrect 
decision.  To  describe  it,  first  let  m  denote  the  maximum  number  of  tests 
which  could  be  conducted  in  any  given  year  (e.g.  m  =  12).  Next,  let 

Lp(k,s)  =  CpP(p  >  pglSj^s)  +  k 

and  L2(k,s)  =  C2E{max(0,po  -  pJJS^  =  s)  +  k  , 

for  k  =  0,...,m  and  possible  values  of  s.  Thus  Lp  and  L2  denote  the 
conditional  expected  losses  far  the  two  decisions,  given  Xp,...,Xk,  plus  the  . 
cost  of  observing  Xp , . . .  ,X^.  If  k  =  0,  then  s  =  0  and  the  expectations  are 
unconditional.  If  sampling  is  terminated  after  k  tests,  then  it  is  optimal  to 
make  decision  1  if  and  only  if  LptkjS^)  <  L2(k,Sjc),  in  which  the  expected  loss 
due  to  terminal  decision  is  * 

/ 

Lg(k,Sfc)=  min{ Lp ( k, Sfc ) , L2 ( k, S^J . 

Let  p(k,s)  =  PtXfc+p  =  1  |  Sk  =  s) 

for  k  =  l,...,m-l  and  possible  values  of  s;  and  define  L  by 
L(m,s)  =  lg(m,s) 

and  L(k, s)  =  min  {Lg(k,s), 


(2)  p(k,s)L(k+l,s+l)  +  (l-p(k,s$L(k+l,s) ) 

for  k  =  0,...,m-l  and  possible  values  of  s.  Then  the  optimal  sampling  plan  is 
to  continue  sampling  as  long  as  L(k,S^)  <  Lg(k,Sj,.),  stopping  at  time 


k 


WTO* 


t  =  minlkXkLgfkjSfc)  =  LO^S^)}. 

Hare  L(k,s)  is  the  minimum  expected  loss  plus  sampling  cost  among  all  sanpling 
plans  which  take  at  least  k  observations. 

If  h  is  a  beta  density,  then  it  is  possible  to  compute  Lj  and  L>2  as  sums 
of  products  of-pft  and  (1-pg)  times  ratios  of  factorials.  I  can  supply  the 
details,  if  you  are  interested.  Using  these  explicit  expressions,  it  is 
straightforward  to  compute  L  by  the  backward  induction  (2);  and,  once  L  and  Lg 
have  been  computed,  it  is  simple  to  classify  the  possible  outcomes  (k,s)  as 
stopping  points,  points  for  which  Lg(k,s)  =  L(k,s),  or  continuation  points. 
Moreover,  the  stopping  points  divide  thens elves  into  lower  stepping  points  for 
which  I^(k,s)  =  L]/k,s)  and  upper  stopping  points  for  which  I^(k,s)  =  L2(k,s). 
If  the  largest  (smallest)  lower  (upper)  stopping  point  is  called  a^  (resp.  fc^), 
then 

t  =  min(k>l:  Sj<a^ar  b^) 

1C  ' 

and  it  is  optimal  to  decide  that  p  <  pg  if  and  only  if  S^-  <  at. 

The  several  tables  which  accompany  this  letter  describe  the  optimal 
sampling  plan  in  a  special  case  in  which  m  =  12,  h^  is  a  beta  density  with 
parameters  a  =  6  and  b  =  2,  w  =  3/4,  pg  =  3/4,  =  60,  and  C2  =  180.  Here  the 

ratio  C2/C1  =  3  equates  the  seriousness  of  deciding  that  p  <  pg  when  p  >  p 
with  that  of  deciding  that  p  >  pg  when  pg  -  p  =  1/3;  and  the  magnitudes  of 
and  C2  were  chosen  to  make  it  optimal  to  take  up  to  about  12  observations. 

I  believe  that  this  is  consistent  with  the  power  and  sample  size  requirements 
discussed  earlier.  __  In  a  certain  sense,  these  values  of  and  C2  are 
implicit  in  those  requirements. 

Table  1  lists  the  boundaries  a^  and  of  the  optimal  test.  These 
boundaries  are  remarkably  insensitive  to  a+b.  I  got  nearly  the  same  values 
when  a  =  9  and  b  =  3.  Table  2  lists  an  ad  hoc  modification  of  the  optimal 
boundaries  which  takes  account  of  th^  economies  of  testing  items  in  groups  of 
three.  Table  3  gives  the  posterior  probability  that  p  >  pg  fear  each  possible 
outcome,  using  the  ad  hoc  boundaries.  It  clearly  exhibits  the  following 
qualitative  feature  of  the  test:  if  the  results  of  the  first  six  tests  this 
year  are  consistent  with  last  year's  results,  then  further  testing  is  not 
optimal.  Table  4  gives  the  frequentist  properties  of  the  adhoc  test,  the  power 
function  and  expected  sample  size  as  a  function  of  p.  Observe  that  the  maximum 
expected  sample  size  is  substantially  smaller  than  that  of  the  adhoc  test;  and 
recall  the  crucial  role  of  the  maximum  in  determining  the  number  of  items  which 
must  be  purchased  for  testing. 


3 

0 

.0251 

6 

1 

.0084 

6 

2 

.0507 

7 

3 

.0813 

8 

4 

.1211 

9 

5 

.1634 

11 

6 

.1185 

12 

7 

.1546 

12 

8 

.3111 

10 

7 

.4543 

8 

6 

.5183 

6 

5 

.6517 

3 

3 

.7450 

TABLE  #4:  FREQUENTIST  PROPERTIES 


£ 

BETA 

MEAN 

VAR 

.05 

.9999 

3.4575 

1.281 

.1 

.  999 

3.8288 

2.4702 

.15 

.9983 

4.4161 

3.5485 

.2 

.9903 

4.8134 

4.5345 

.25 

.9788 

5.4154 

5.43 

.3 

.9582 

5.8102 

6.2305 

.35 

.9244 

6.3797 

6.8348 

.40 

.8728 

6.7887 

7.559 

.45 

.8000 

7.1384 

8.1442 

Comments  by  DW : 


With  this  note  Woodroofe  completes  the  transition  from  Wald's 
classic  treatment  to  a  Bayesian  approach.  The  use  of  a  prior 
probability  which  is  a  mix  of  two  hypotheses  is  in  part  an  attempt  to 
address  the  criticism  that  priors  can  become  too  sharply  peaked, 
neglecting  the  potential  staleness  of  old  data.  One  might  still  ask 
whether  there  should  be  an  upper  limit  to  the  value  of  k  used  in  the 
prior . 

The  loss  functions  included  in  this  section  are  representative, 
rather  than  my  recommendation.  The  variable  called  po  in  the 
functions  LI  and  L2  could  have  different  values  in  the  two  cases. 


Chapter  V 

Other  Stopping  Criteria 


A  possible  argument  for  small  test  sizes  may  arise  after  all 
missiles  have  been  bought:  any  test  reduces  the  potential  tactical 
inventory.  The  decision  criterion  is  unfortunately  not  unique.  This 
chapter  discusses  a  few  examples. 

Section  1.  Utility  as  a.  Criterion 

Let  4  (?  i  be  the  posterior  probability  distribution 

of  p,  given  s  "equivalent"  successes  and  f  "equivalent"  failures  on 
which  to  base  a  prediction.  Let  U  (N,p)  be  the  "utility"  of  an 
inventory  of  N  missiles  of  reliability  p.  The  estimate  of  the 
utility  of  the  inventory  is  then 

0(n)  =  4  (ns>fVp 

Now  perform  a  test:  N  goes  to  N-l;  with  probability  p, 
s  goes  to  s+1;  and  with  probability  1-p,  f  goes  to  f+1. 

After  the  test  the  utility  is 

u( M-  \)  =  \  UCkH* 

The  criterion  is:  Is  U(N-1)>U(N)? 


Examples  of  utility  functions  are: 
Np  (expected  targets  killed); 


-Np(l-p)  (uncertainty  is  reduced); 


N-T/P  (excess  inventory,  where  T  is  size  of 
target  list); 

t[i-  o  -0N/r3  (expected  damage) ; 

tO  (b=largest  integer  in 

fractional  part;  this  reduces  to  Np  for  small  N, 
damage  for  large  N). 


critical 


N/T;  a=N/T-b  is  the 
goes  to  expected 


Clearly  there  is  a  similarity  between  this  method  and  that  in 
Secion  4  of  the  previous  chapter. 


Section  2 .  Information  as  £  Criterion 

Another  criterion  would  be  the  information  the  decision  maker 
gains  from  the  test  about  the  posterior  distribution  of  p.  This 
would  be  applicable  when  no  single  utility  function  can  be  agreed  on. 
An  example  is  the  Kullback-Leibler  information  measure  on  two 
probability  density  functions 


FI  and  F2  (Reference  18): 


r(?,  ^  ■ 

It  can  be  applied  to  the  current  problem  by  defining  FI  and  F2 
respectively  as  the  posterior  and  prior  density  functions  for  p. 

Shannon's  information  measure  S(F1,F2)  is  the  expectation  value  of 
I(F1,F2)  over  the  observed  values  of  success  and  failures. 

To  illustrate,  we  may  identify  F2  with  expression  1.6  from 
Chapter  I: 

and  FI  with  expression  1.8: 

so  that  log  F1/F2  is  — i 

».(l. ti.\ 


-  C*  +  "fv  'f) 


where  C  is  the  logarithm  of  the  gamma-function  combination  in  curly 
braces,  all  independent  of  p.  Noting  that 

h/  \  J_ 

and  letting  tC«)2  rfr)  At  the  logarithmic  derivative  of  the 
gamma  function,  the  expression  for  I(F1,F2)  reduces  to 


T  (f,  ft)  =  C  -  ^  (s,  «•  ^  ^  6,+^v)  -4(v  Q  ] 


Then 


Consider  now  the  case; where  s2=n2=l  (a  single  successful  trial) 


In  the  alternative  case  wher  S2«=0,n2=l  (a  single  unsuccessful  trial) 


^  0+  *)  -vv C >  C )] 

and  the  Shannon  information  is  S  \  A.e'h  f.rF  i  _ 

s  -  - 3 - — ■  Ci  —  r  *  •  * 

^  ^ .  2.n , 


As  this  never  goes  to  zero  (for  finite  nlj,  the  cost  of  this 
information  must  be  balanced  against  the  use  made  of  it. 


I  have  not  yet  found  a  way  to  apply  this  criterion  to  the 
Pershing  testing  problem. 


Chapter  VI 
Conclusion 

I  return  now  to  the  tasking  from  the  Under  Secretary  of  the 
Army,  as  given  in  the  opening  of  this  memorandum.  The  mathematical 
methods  of  sequential  analysis  proposed  here  for  estimating 
reliability  changes  possess  a  rigor  not  found  in  the  Army's  current 
method,  and  make  clear  the  risks  in  following  their  prescription. 

They  provide  a  basis  for  reducing  the  size  of  an  annual  test  and  so 
reducing  too  the  cost  of  a  testing  program.  Indeed,  they  even 
challenge  the  need  for  an  annual  report,  and  suggest  that  the 
interval  between  reports  can  be  enlarged  (e.g.,  to  two  years)  with  no 
increase  in  risk  to  management.  They  do  not,  however,  encompass  a 
variety  of  other  issues  which  are  fundamentally  operational  in 
nature:  firings  to  support  training,  alternate  uses  of  inventory, 

system  life.  These  must  be  the  subject  of  further  investigation. 

Readers  of  this  report  may  be  disappointed  that  such  very 
different  approaches  to  the  stopping  problem  have  been  presented  in 
the  foregoing  chapters.  I  observe  that  such  a  seemingly  simple 
problem  has  apparently  not  been  hitherto  subject  to  the  scrutiny  it 
deserves,  and  that  it  is  comforting  that  two  separate  investigations 
have  reached  similar  conclusions. 

I  see  ultimately  more  promise  in  the  methods  proposed  in  Chapter 
IV,  but  would  recommend  that  those  of  Chapters  III  and  IV  be  applied 
to  Pershing  using  the  best  available  data  so  that  a  refined  test 
program  can  be  determined.  In  Chapter  III  is  proposed  the 
application,  as  yet  unexplored,  of  Kalman  filtering  techniques  to 
this  problem.  This  research  merits  monitoring,  if  not  support. 
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(l/)  1  ilC  various  ;  .  ..uMj.li-ms  r<..|Ui*e,!  in  ii  =  -  '  ;  i  ;i!.  ('  ■:>  of  the  i r,;.i h  to  - *s 
of  the  '•■i.'u'c)  be  «-;>o  :.;.Y  ihc  1  •  the  italics  a  i <  'her  data  j'u:c\ :  i:  a  ihvci>..i  m 
d.  riving  numerical  ;>v .  fci....m>.'c  c.tir.;  \  *■  firm  the  to*  >:■::■  ‘.hoi.U  Ik  Jvhr.d  h,i 

t.;h  perfon  ii.vu.it.  •v.'ud  is*,  ih.  i  1  ]h  c  in  liiv  ta!:  .nations  should 

be  summari/i  d  !•->  pvirnii  verification  of  t  lie  analytical  ap-pio;-;  h. 

H.  fiE.N5ITl\  iTY  ANALYSIS 

(U)  A  sensitivity  analysis  fhould  ba  conducted  for  i acli  peifo«r\-::.<.e  '•c*i:  .;..c  to 
...tv  v  ’..-vib.ur  i.uc  r.uv..  rival  results  woo'd  chance  sii'ruix rnliy  if  Ihc  tt-ainicr.!  of  test  01 
data  anomalies  wore  changed. 

F.  COXriDLNCE  STATEMENTS 

(U)  Two  types  of  confidence  statements  should  be  provided  for  each  performance 
factor: 

(1)  A  statistical  confidence  bound  based  upon  the  quantity  of  data  used  in 
compsiting  the  factor. 

(2)  A  qualitative  assessment  based  upon  the  quulity  of  data  used  in  computing  the 
factor. 

Hie  qualitative  assessment  should  be  based  upon  an  appraisal  of  the  validity  and  applicabil¬ 
ity  of  the  test  data  as  outlined  in  Part  1  of  these  guidelines. 

(U)  The  statistical  significance  of  differences  in  estimates  of  performance  factors  that 
is  indicated  by  comparisons  of  the  results  of  different  sets  of  Operational  l  est  data  should 
be  addressed  and  statistical  confidence  statements  regarding  these  differences  should  be 
provided.  The  results  of  one  method  for  comparing  reliability  samples  is  illustrated  in  Table  4. 
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Appendix  D 
HP-41  Programs 

The  HP-41  handheld  calculator  is  slow  but  remarkably  powerful. 
For  example,  a  program  listing  for  the  standard  Fast  Fourier 
Transform  (FFT)  algorithm  is  no  lengthier  than  that  for  a  FORTRAN 
version  and  because  of  some  quirks  of  the  HP-41,  the  program  is  in 
some  ways  more  efficient.  With  a  56-bit  word,  numerical  accuracy  is 
higher  than  in  most  personal  computers,  and  so  round-off  problems  are 
slower  to  arise. 

Reported  in  this  appendix  are  a  set  of  programs  written  for  this 
study.  Their  original  purposes  were  to  give  or  to  verify  solutions, 
but  they  have  two  additional  values  justifying  their  inclusion  here: 
they  demonstrate  that  the  mathematics  called  upon  is  not  intractible 
and  can  be  packaged  small,  and  they  may  be  useful  as  is  to  others 
working  the  same  or  related  problems. 

The  first  group  provde  solutions  to  Equations  1.9  and  1.11  and 
thus  can  be  considered  a  proper  means  of  getting  the  answers  wrongly 
sought  via  Fisher's  Exact  Test.  The  versions  given  are  lengthy  but 
are  relatively  robust  to  the  accumulation  of  round-off  errors. 
Included  is  the  program  PII,  written  to  be  a  model  for  and  to  verify 
calculations  of  Singpurvalla  and  Launer. 

The  second  group  provide  handy  means  of  exploring  Woodroofe's 
treatment  of  sequential  analysis.  ET  provide  solutions  to  Equations 
1  and  2  of  Chapter  III,  Sec  1.  BND  provides  Wald's  and  Woodroofe's 
boundaries  of  the  region  of  test  continuation;  and  MW  permits 
computation  of  a  number  of  properties  of  a  test  plan  defined  by  BND. 
LOP  computes  boundaries  using  the  Bayesian  method  of  Chapter  III, 

Sec .  4 . 

Not  included  is  a  package  of  routines  which  manipulate  truncated 
Taylor  series  and  was  used  to  compute  the  expansion  of  D(n,m)  given 
in  Eq  4.4.  This  is  available  from  the  author. 

The  memory  requirements  of  an  HP-41CV  or  CX  are  needed,  and  if 
it  is  not  the  CX  version,  then  an  Extended  Functions  module  (XF)  with 
its  Expanded  Memory.  The  occasional  use  of  Synthetic  Programming  can 
be  circumscribed,  t  if  the  programs  are  identical  to  those  listed 
here,  they  should  run  on  any  version  of  the  HP-41  with  adequate 
memory  and  the  XF  module. 


JCS+ 


Implements  Eq.1.9  and  DA+  Eq.1.11. 


They  call  for  inputs  and  report  the  value  of  the  integral  as 
"CL  =  "  for  Confidence  Level.  The  plus  sign  means  there  are  no 
subtractions  in  the  algorithm,  hence  less  round-off  error. 

PII  Implements  Eqs.4-6  of  Section  III. 3. 


Entering  at  LBL  A  leads  to  an  evaluation  of  and  at  LBLT£>  to 
evaluation  of  (3  .  Lines  51-62  clear  a  block  of  registers,  using 

program  BC  in  a  module  called  PPC  ROM.  This  can  be  replaced  by 
ordinary  coding.  If  Flag  02  is  set,  then  the  summation  sign  in  Eq.4 

or  5  is  ignored;  only  a  single  term  is  considered.  Subroutines  1,  2, 
and  13  are  the  core  of  algorithm. 


ET 


Solves  Eqs.  1  and  2  of  Section  IV. 1. 

(WO-rf4 

j  M 


and 
4~  fJ-ro  -  f 


I 


t  (0=2^ (<-■) f  0- 0  c  -v  2T  ««  (&)r  *'"•'(- 

Calls  for  N,  c,  and  p  (unadjusted  values  will  be  used  as  is). 


Memory  utilization  keyed  to  that  in  MW:  N,  c,  and  p  in  same 
registers . 

MW 


Requires  two  files  in  Extended  Memory  named  Am  and  Bm  where  m  is 
a  number  provided  in  response  to  query  "FILE#?”  or  is  already  stored 
in  register  19.  (Routine  BND  may  have  been  used  to  create  these 
files .  ) 

Start  program  at  line  1  or  at  LBL  E;  line  one  to  provide/revise 
the  value  of  N,  the  maximum  number  of  tests.  At  E,  provide  "p"  and 
"FILE#."  If  RAD-DEG  selection  set  to  RAD,  program  computes  and 
reports  G(k)  as  required  by  Section  IV. 3;  if  set  to  DEG,  this  is 
ignored . 

Program  reports  ^  (p)»  E(t),  and  a  (p)  (which  in  effect 
interchanges  meaning  of  "reliable”  and  "unreliable*).  Sect  IV. 1. 


LBL  B  produces  output  stating  "bi/i  =  cumulative  probability  of 
sufficient  failures  to  halt."  Accumulates  probability  of  exit 
passing  clockwise  around  boundary.  If  there  are  several  points  on 
boundary  at  N=N  max,  then  these  are  labeled  F.  Then  program 
continues  along  "a"  boundary. 

LBL  C  does  the  same  as  LBL  B  but  counterclockwise. 

LOP 

To  meet  the  goals  of  Section  IV. 4.  Computes  the  boundary 
conditions  for  continued  testing,  based  on  the  loss  functions  LI  and 
L2  (which  can  have  associated  with  them  different  criteria  PI  and  P 2 , 
as  well  as  cost  factors  Cl  and  C2). 

Program  invites  all  necessary  input  insertion/revision/ 
verification,  and  then  constructs  a  diagram  of  the  operating  space. 

To  conserve  space  this  pattern  is  stored  as  packed  binary  data  (a  la 
flags).  LBL  J  provides  a  visualization  of  this  pattern,  for  display 
or  printing  (see  figures  below).  This  algorithm  has  also  been  run  on 
a  Commodore  for  verification. 

Routines  6  and  7  support  generation  of  loss  functions  L/  and  L^ 
If  others  are  chosen,  these  must  be  rewritten  along  with  some  of 
Routine  2  (lines  57-100). 

BND 

Develops  the  boundaries  to  be  used  in  MW,  by  Wald's  and 
Woodroofe's  methods.  Input  called  for:  P0,  PI,  a,  and  b  (later,  m). 

0<P0<P1<  1.  Level  of  test  =  a.  Probability  of  Type  II  error  **  b 

(P^-  PI).  Ho:  p  <  po .  (Section  IV. 2).  M  is  number  of  tests. 

Lines  1-85:  Wald's  methods,  an*  and  b  r/  reported  out. 

86-156:  Woodroofe's  modification. 

157-END:  Subroutine  E.  Calls  for  a  file  number  k;  then  stores 

Woodroofe's  boundary  numbers  a  rv  and  b^  in  files 
AK  and  BK.  If  Flag  25  is  clear  to  start,  program 
halts  if  attempt  is  made  to  overwrite  existing  file. 
Set  the  Flag  to  permit  overwriting. 


0 1  ♦LBL  “JOS’ 

5 1 ♦LBL  81 

96*LBL  83 

82  CF  2? 

52  RCL  86 

97  RCL  11 

03  "DEL=“ 

53  STO  67 

98  RCL  88 

84  SF  80 

99  YtX 

85  . 

544LBL  82 

108  ST*  12 

06  XEQ  88 

55  RCL  86 

181  RCL  82 

87  *Hl=’ 

56  RCL  07 

102  E 

88  E 

57  - 

103  - 

80  XEQ  86 

58  LflSTX 

184  RCL  88 

59  E 

185  + 

10*LBL  B 

68  - 

106  LflSTX 

11  ’Sl=* 

61  / 

187  XEQ  84 

12  2 

62  ?CL  10 

108  $!♦  12 

13  XEQ  06 

63  RCL  07 

109  RCL  01 

64  - 

118  E 

14 ♦LBL  C 

65  LflSTX 

111  - 

15  “N2=* 

66  RCL  09 

112  RCL  88 

16  3 

67  + 

113  + 

17  XEQ  ee 

68  / 

114  LflSTX 

69  * 

115  XEQ  84 

18*LBL  P 

78  RCL  00 

116  ST/  12 

18  m$2-‘ 

71  / 

117  *CL=’ 

20  4 

72  E 

118  FIX  4 

21  XE6  00 

73  X<>  13 

119  flRCL  12 

74  ♦ 

120  fiVIEH 

22*LBL  18 

75  ST+  13 

121  STOP 

23  *REL  DEG- 

76  ISG  87 

122  RTN 

24  fiVIEH 

77  GTO  82 

25  RCL  80 

78  RCL  88 

123*L6L  80 

26  CHS 

79  CHS 

124  FIX  8 

27  E 

88  RCL  06 

125  FS?C  06 

28  + 

81  - 

126  FIX  4 

29  STO  11 

82  LflSTX 

127  flRCL  I HD  X 

38  RCL  84 

83  E 

128  PROMPT 

31  E 

84  - 

129  FS?C  22 

32  + 

85  / 

130  STO  IND  V 

33  RCL  83 

86  RCL  80 

131  RTN 

34  - 

87  RCL  11 

35  STO  05 

88  / 

1324LBL  04 

36  STO  06 

89  * 

133  CHS 

37  LflSTX 

98  RCL  13 

134  XOY 

38  E 

91  X<>  12 

135  SIGN 

39  - 

92  * 

136  X<>  L 

49  STO  83 

93  ST+  12 

137  ST+  Y 

41  E 

94  ISG  86 

42  - 

95  GTO  81 

1384LBL  85 

43  RCL  82 

139  X=Y? 

44  + 

146  GTO  06 

45  STO  89 

141  ST*  L 

46  IfiSTX 

142  DSE  X 

47  CHS 

143  GTO  05 

48  RCL  8! 

49  ♦ 

1 44 ♦ LBL  86 

50  STO  18 

145  PI'S 

146  t:  l 

147  FTN 

146  .ENT. 

JOA+ 


I 

ft 


IvJj! 

Yi 

$$ 


+- 

01*LBL  “DA*" 

48*L8L  81 

49  RCL  06 
58  STO  07 

51  RCL  03 

52  XOY 

53  + 

02  CF  29 

54  STO  10 

63  SF  06 

55  LflSTX 

04  -CEL=- 

56  E 

85  . 

57  + 

06  XEQ  08 

58  RCL  85 

07 *LBL  A 

59  ♦ 

68  STO  12 

08  "NO- 
09  E 

61 ♦LBL  02 

1@  XEQ  00 

62  RCL  12 

U*LBL  B 

63  RCL  87 

64  - 

12  -SO* 

65  STO  13 

13  2 

66  RCL  62 

14  XEQ  00 

6?  E 

15*LBL  C 

68  - 
69  CHS 

16  *N2=* 

70  STO  08 

17  3 

18  XEQ  09 

71*LBL  03 

19*LBL  D 

72  E 

73  RCL  82 

20  *S2=- 

74  - 

21  4 

75  RCL  88 

22  XEQ  00 

76  - 

23*LBL  10 

77  LASTX 

78  E 

24  -ABS  DEG* 

79  - 

25  AVI  EH 

80  / 

26  RCL  00 

81  RCL  10 

27  1/X 

82  RCL  88 

28  E 

83  - 

29  - 

84  LflSTX 

38  STO  09 

85  RCL  13 

31  RCL  81 

86  XOY 

32  RCL  82 

87  - 

33  - 

88  / 

34  STO  11 

89  * 

35  RCL  03 

98  RCL  89 

36  + 

91  * 

37  E 

92  E 

38  - 

93  X<>  14 

39  STO  05 

94  * 

48  RCL  84 

95  ST+  14 

41  RCL  03 

96  ISG  83 

42  - 

97  GTO  63 

43  E 

98  RCL  86 

44  ♦ 

99  RCL  07 

45  STU  86 

188  - 

46  . 

47  STO  16 

101  LflSTX 

102  E 

103  * 

104  / 

105  RCL  11 

106  RCL  0? 

107  - 

108  LfiSTX 

109  RCL  12 

118  XOY 
111  - 
112  / 

113  * 

114  RCL  89 

115  * 

116  RCL  14 

117  XG  15 

118  * 

119  ST+  15 

120  ISG  07 

121  GTO  02 

122  RCL  05 

123  CHS 

124  RCL  06 

125  - 

126  LflSTX 

127  E 

128  - 

129  / 

130  RCL  09 

131  / 

132  RCL  15 

133  X<>  16 

134  * 

135  ST+  16 

136  ISG  86 

137  GTO  01 


138*LBL  04 

139  RCL  80 

140  RCL  02 

141  E 

142  - 

143  YtX 

144  ST*  16 

145  RCL  88 

146  CHS 

147  E 

148  + 

149  RCL  85 
158  YtX 


151  ST*  16 

152  RCL  01 

153  E 

154  - 

155  RCL  X 

156  RCL  02 

157  E 

158  - 

159  - 

160  XEQ  85 

161  ST*  16 

162  RCL  05 

163  RCL  X 

164  RCL  03 

165  E 

166  - 


168  XEQ  85 

169  ST/  16 

170  FIX  4 

171  -CL=’ 

172  flRCL  16 

173  ftVIEH 

174  BEEP 

175  STOP 

176  RTH 


177*LBL  05 

178  CHS 

179  XOY 
188  SIGH 

181  X<>  L 

182  ST+  Y 


183*L6L  86 

184  X=Y? 

185  GTO  07 

186  ST*  L 

187  DSE  X 

188  GTO  06 


189*LBL  07 

190  RDN 

191  X<>  L 

192  RTH 


193*LBL  08 

194  FIX  8 

195  FS?C  80 

196  FIX  4 
197  flRCL  INS  X 

198  PROHPT 

199  FS’C  22 
200  STO  INS  v 

281  RTN 
202  END 


ViwS. 

Vc>C 


Lc 


<vsro  \tn>Y 

o<o  btaj 

f-7  Lei  Of 
0Sf*?C  ol 
C^sFO) 

;6  )<>  03 

11  >  ,  ;S 

It  XO  63 
13  RC.  !6 
H  RCL  !7 

15  - 

16  E  '• 

17  - 

18  STO  ,7' 
l?  r.I'N 

86  RTH 

21  ♦LBL  ft 

22  SF  06 

23  GTO  16 

244LBL  B 
25*L6L  -PI1* 
26  CF  60 

27*LBL  16 

28  CF  61 

29  FIX  2 
36  CF  22 

31  ”BEL=‘ 

32  18 

33  XEQ  06 

34*LBL  C 

35  'GfiHHfl=' 

36  3 

37  XEQ  08 

38  ’BELTP=‘ 

39  15 

48  XEQ  60 

41 ♦LBL  B 

42  FIX  8 

43  "H=* 

44  16 

45  XEQ  06 

46*LBL  E 

47  FIX  0 

48  *X=“ 

49  17 

50  XEQ  06 

51  RCL  17 

52  RCL  16 

53  E 

54  - 

55  £3 

56  / 

57  FS?  62 

58  XOY 


I^UdLCVeor  76.4: 

63  Ru  IE 

64  F-. '  66 

65  Chi 

66  X/0? 

67  XEQ  89 

68  ftSS 

69  STO  99 

76* LBL  67 

71  RCL  19 

72  IHT 

73  STO  17 

74  RCL  89 

75  X=0? 

76  GTO  16 

77  Es'TEFf 

78  CHS 

79  E 
86  + 

81  STO  16 

82  / 

83  STO  68 

84  -  E 

85  RCL  15 

86  + 

87  STO  14 

88  LftSTX 

89  RCL  16 

90  RCL  63 

91  + 

92  STO  04 

93  + 

94  STO  65 

95  . 

96  STO  13 

97  RCL  16 

98  STO  60 

99  x=e? 

106  GTO  20 

18ULBL  61 

102  XEQ  21 

103  RCL  16 

184  E 

185  + 

106  RCL  14 
16?  RCL  64 

108  RCL  08 

109  ST-  T 
118  ST+  Z 

111  ST-  Y 

112  * 

113  / 

114  * 

115  RCL  69 

116  / 

117  ST*  13 

118  BSE  66 

119  GTO  01 


Ol«c U  \/t- 


123  RC L  16 

124  YtX 

125  F.Cl  16 

126  RCL  83 

127  RCL  14 

128  + 

129  YtX 

130  * 

131  ST*  13 

132  RCL  64 

133  E 

134  - 

135  RCL  16 

136  XEQ  64 

137  ST*  13 

138  RCL  65 

139  E 
148  - 

141  RCL  16 

142  XEQ  04 

143  ST/  13 

144*LBL  17 

145  XEQ  19 

146  RCL  12 

147  ST/  13 

148  FIX  4 

149  FC?  00 
158  *b=- 

151  FS?  80 

152  "l-a=‘ 
■153  E 

154  RCL  13 

155  FC?  01 

156  - 

157  flRCL  X 

158  ‘F  X=* 

159  FIX  6 
166  RCL  17 

161  flRCL  X 

162  20 

163  + 

164  XOY 
165  STO  INB  Y 

166  flVIEH 

167  ISG  19 

168  GTO  67 

169  FS?  61 
176  XEQ  69 

171  BEEP 

172  STOP 

173*LBL  J 

174  28.02 

175  RCL  19 

176  FPC 

177  ♦ 

178  XROK  20/67 
179  RTN 


182  X  • 

L 

c>c 

r  i.  L 

H, 

305  Si'  13 

36? 

RCL  16 

183  X-0 ' 

243 

RCl 

01 

306  GTO  1? 

368 

E 

184  GTO 

66 

244 

ST- 

Y 

369 

4 

185  X<  ;'i 

245 

/ 

307*LBl  19 

376 

RCL  01 

246 

4 

368  RCL  63 

371 

ST-  Y 

1 56* LBL 

65 

24? 

rCL 

88 

309  t 

372 

/ 

IS?  ST» 

L 

248 

/ 

310  STO  11 

373 

R1 

188  BSE 

X 

249 

E 

311  - 

374 

4 

189  " 

256 

+ 

312  STO  02 

375 

* 

198  BSE 

Y 

251 

DSE 

61 

313  X=9? 

376 

E 

191  GTO 

85 

252 

GTO 

63 

314  GTO  92 

377 

4 

192*L8L  66 

193  RBN 

194  X<>  L 

195  RTH 


196*LBL  21 
19?  RCL  16 

198  RCL  66 

199  - 

286  STO  6? 

261  RCL  17 

262  X>Y? 
2e3  XOY 

264  STO  81 

265  E 
286  ST+  67 
267  RCL  64 

288  RCL  66 

289  - 

216  STO  06 
21!  RCL  63 

212  E 

213  STO  12 

214  - 

215  STO  62 

216  X=6? 

217  GTO  15 


218*LBL  02 
219  RCL  65 

228  RCL  63 

221  RCL  66 

222  RCL  62 

223  ST-  T 

224  ST-  Z 

225  ST-  Y 

226  ♦ 

227  / 

228  * 

229  RCL  68 
236  * 

231  ST*  12 

232  E 

233  ST+  12 

234  DSE  82 

235  GTO  62 


236*LBL  15 
23?  RCL  61 

238  X=0? 

239  GTO  14 

'■AO  C 


254*LBL  14 

255  RCL  12 

256  ST+  13 

257  RTH 


258*L£l  16 

259  CF  61 
268  RCL  16 

261  E 

262  STO  13 

263  + 

264  STO  64 

265  RCL  63 

266  E 
26?  - 

268  STO  85 

269  RCL  16 
276  RCL  15 

271  ♦ 

272  STO  86 

273  RCL  17 

274  STO  66 

275  X=0? 

276  GTO  18 


27?*LBL  13 

278  RCL  64 

279  RCL  85 

286  RCL  66 

281  RCL  66 

282  ST-  T 

283  ST+  Z 

284  ST-  Y 

285  * 

286  / 

287  * 

288  ST*  13 

289  E 
296  ST+  13 

291  BSE  68 

292  GTO  13 


293*LBL  18 

294  RCL  14 

295  RCL  16 

296  ♦ 

297  LftSTX 

298  XES  64 

299  ST*  13 
306  RCL  65 
361  RCL  86 


315  RCL  69 

316  X=8? 
31?  GTO  92 

318  RCL  15 

319  E 


321 ♦lBl  91 

322  ENTERt 

323  EHTERt 

324  RCL  88 

325  * 

326  RCL  62 

327  / 

328  E 

329  X<>  11 
336  * 

331  ST+  11 

332  RUN 

333  1SG  X 

334  ” 

335  BSE  82 

336  GTO  91 


337*LBL  92 

338  RCL  69 

339  CHS 
346  E 

341  + 

342  RCL  14 

343  RCL  63 

344  + 

345  YtX 

346  RCL  11 
34?  * 

348  STO  12 

349  RTH 


356*LBL  'PI- 
35  1  STO  61 

352  RCL  15 
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APPENDIX  A 


An  Algorithm,  A  Computer  Code,  and  A  User's  Guide,  for 
a  Bayesian  Binomial  Hypothesis  Testing  Procedure 


A.l.  INTRODUCTION 

In  the  Bayesian  binomial  hypothesis  testing  procedure,  we  need 
to  find  the  pair  (nt>x*)  such  that  [see  Equations  (4)  and  (5)]: 

1  x* 


J  ^ 

Jo  J=° 


n 

t 

U  J 


n  “j 

Pt  ^  Z  8(Pt)dpt  <  a 


and 


1  x*  r 

/t  nr  n  -j 

I  (Pr  -  A)J  (1  -  p  +  A)  C  g(p  )dp  >  1-6 

j=0  l j  J  c  c  c 


where 


,  _  T(y+5)  y-1  ,6-1 

8(pt)  r(y)r(6)  pt  (1  '  Pt} 


The  above  inequalities  can  be  rewritten  as: 

„(*,„)  ,  r(r+*>  y 

t  t,ntJ  r(Y) r(6) 


V 


U  J 


r(j+Y)r(nt-j+6) 

r(nt+Y+6) 


g  (x*  n  )  =  r<X+(S)  Ant  f  I  t 

g2ut,nt;  r(Y)r(6)  A 


j  fjl 

_  £=0  {  J 

A“£(-1)J  z 


V3  fn  — j I 

-  v  -  / 

r 

(9A) 

I  1 

A  m  B(A,1;  £+6,  m4-6) 

>  1  -  6  , 

where 


1 

B(A,1;  r ,  s)  =  f  p^"1  (1  -  P,.)3"1  dP|.  . 

A 

A  computer  code  designed  to  obtain  the  smallest  values  of  n  , 
x*  subject  to  the  two  inequalities  (8A)  and  (9A) ,  based  on  an  enumera¬ 
tion  procedure  discussed  next,  is  obtained. 

A. 2  DESCRIPTION  OF  THE  ENUMERATION  PROCEDURE 

The  enumeration  procedure  exploits  the  fact  that  both  g  (x  ,n  ) 
and  82^xt»nt)  are  increasing  functions  of  x^  if  n^  is  fixed.  The 
procedure  starts  with  some  initial  value  of  n^  ,  say  n^  ,  and  finds 
the  largest  xc  such  that  8^(xt»n^)  ^  a  •  Once  such  an  xfc  ,  say  x^ 
is  found,  it  is  guaranteed  that  the  first  inequality  will  be  satisfied 

for  values  of  x  smaller  than  x^  .  The  procedure  then  tries  to 

find  an  x  smaller  than  x^  such  that  g2(xt,nt>  ^  1  -  8  .  If  such 

an  xfc  does  not  exist,  the  value  of  n^  is  increased  by  one  and  the 

procedure  starts  all  over  again.  As  n^  increases,  the  procedure  finds 
the  smallest  values  of  nt  and  x  satisfying  both  inequalities.  The 
flow  chart  for  this  enumeration  procedure  is  presented  in  Figure  A.l. 


A. 3  THE  COMPUTER  CODE 

The  program  requires  certain  JCL  cards  and  a  user  input  of  some 
parameters . 


The  cards  should  be  arranged  in  accordance  with  Figure  A. 2;  each 


card  will  be  explained  individually. 

Job  Card  and  JCL  Cards:  The  standard  job  card  is  used  and  so  are 
the  following  JCL  cards: 

//J6EXECJ6FORG2 
//FORT . SYSINfe5DD 
//GO.SVSLIBtzSDD 

//0M0J6DD0WiJ0DSN=GWU .  IMSL . V9 .DLOAD,DISP=SHR 
//G0.SYSINWJW5DDWWJJ* 

where  the  character  "J5"  indicates  a  blank  space.  The  first  two  JCL 
cards  immediately  follow  the  job  card.  The  remaining  JCL  cards  are 
placed  after  the  program  and  just  before  the  input  information  card. 

The  fourth  JCL  card  is  needed  to  use  the  IMSL  subroutines  on  an  IBM 
machine . 

Input  Information  Card — DEL,  SGM,  SDEL,  ALF,  BETA,  NT:  This  card 
contains  sorted  input  information,  DEL,  SGM,  and  SDEL,  which  are  the 
parameters  A  ,  y  ,  and  6  in  Equations  (8A)  and  (9A) ;  ALF  and  BETA  are 
the  right-hand  side  parameters  a  and  (3  in  these  inequalities.  These 
parameters  are  specified  in  format  F10.5.  The  input  NT  is  the  initial 
value  of  n  selected,  and  is  in  14  format.  Usually,  this  value  is  one, 


A. 3. 2  Interpretation  of  Output 

The  program  uses  an  iterative  scheme  and  evaluates  g^(xt,nt) 
and  g2(xt»nt)  for  different  values  of  adn  nt  .  On  the  output, 

the  values  of  g^(xt,nt)  and  g2(xc,nt>  are  printed  as 

FIRST  CONST  = 

SECOND  CONST  = 

for  different  values  of  xt  and  n^  . 

The  solution  of  the  problem,  that  is,  the  smallest  values  of  x 


and  nt  satisfying  the  inequalities  (8A)  and  (9A) ,  are  printed  in  the 
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last  line  of  the  output  as 

X  =  N  = 

Sample  output  is  presented  in  Table  A.l. 

The  smallest  values  of  x^  and  nt  satisfying  the  inequalities 
(8A)  and  (9A)  are  X  =  10  and  N  =  15  .  In  this  example,  the  values 
of  the  parameters  are  A  =  0.25  ,  y  =  106  ,  6  =  19  ,  a  =  0.10  ,  and 

8  =  0.25  .  The  initial  value  of  n  is  one. 

The  listing  of  the  program  is  given  in  Appendix  B. 


L  I  M=  1 , 5 
.(-TEST 

//  EXEC  F0RX2 
//FORT. SYS  IN  OD  * 

IMPLICIT  REAL*8  (A-H.O-Z) 

INTEGER  IER 

REAO (5,10)  DEL.SGM.SDEL.ALF.BETA.NT 

10  FORMAT  (5F 10.5. HO 
BET-1 .O-BETA 

X  1-DEL 
X2-  1  .0 

C  WE  START  THE  ALGORITHM  BY  INITIATING  XT  AS  ZERO 
W1=SGM 
W2-SDEL 
Al-Wl 
8 1=W2 

CALL  F ACT  1  (Al.Bl.SON) 

W-SON 

11  XT=0 .0 
WNT=NT 
WL=WNT+$DEL 
TA1-SGM 

TB 1 -WL 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

PAR=TERS 

C01-W*PAR 

C  THIS  IS  THE  VALUE  WHEN  XT  IS  ZERO 

C  NOW  WE  COMPUTE  THE  VALUE  G1  WHEN  XT  IS  OTHER  THAN  ZERO. 

301  IXT-XT 
T0T-C01 

IF  (XT.EQ.O.O)  GO  TO  1001 
DO  1000  1-1. IXT 
R I  *  I 

Pl-Wl+R I 
P2-WL-RI 
TA1-P1 
TB1-P2 

CALL  FACT2  (TA1 ,TB 1 ,TERS) 

P3-WNT+1 .0 
PL-P3-RI 

P5-RI+1 .0 

Z=  (DGAMMA  <P3) )  /  ( (DGAMMA  (PL) )  *  (OGAMMA  (P5) ) ) 

P-TERS 

TOT-TOT+ (P*Z*W) 

1000  CONTINUE 

1001  G 1-TOT 

WRITE  (6, 60)  Gl.XT.NT 

60  FORMAT  (5X, 'FIRST  CONST-' , F 10 , 5 . 5X, ' XT- ' , F5 . 1 , 5X , ' NT-  . lO 
C  SO  WE  COMPUTED  THE  VALUE  OF  FIRST  CONSTRAINT 
IF  (G 1 .GT.ALF)  GO  TO  333 
IF  (XT.EQ.KT)  GO  TO  360 
XT-XT+1 .0 
GO  TO  301 
333  XT-XT-1.0 


A192  136  PERSHING  II  EOLLON-ON  TEST'  SIZE  REDUCED  BV  SEOUENTIAL 
ANALVSI5(U>  DEPUTV  UNDER  SECRETARV  OF  THE  ARHV 
(OPERATIONS  RESEARCH)  UASHINGTON  DC  D  WILLARD  ET  AL 
UNCLASSIFIED  B1  SEP  84  F/G  16/4 


LIM-1,5 

.l-TEST 

//  EXEC  F0RX2 
//FORT. SYS  IN  DO  * 

IMPLICIT  REAL*8(A-H,0-Z) 

INTEGER  IER 

READ  (5, 10)  DEL,SGM,SDEL,ALF , BETA.NT 

10  FORMAT  (5F  10.5.  I1*) 

BET-1. O-BETA 
Xl-DEL 

X2-1 .0 

C  WE  START  THE  ALGORITHM  BY  INITIATING  XT  AS  ZERO 
W1 =SGM 
W2-SDEL 
Al-Wl 
B1-W2 

CALL  FACT1  (A1.B1.S0N) 

W-SON 

11  XT-0 .0 
WNT-NT 
W4-WNT+SDEL 
TA1-SGM 
TB1-WL 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

PAR-TERS 

C01-W*PAR 

C  THIS  IS  THE  VALUE  WHEN  XT  IS  ZERO 

C  NOW  WE  COMPUTE  THE  VALUE  G1  WHEN  XT  IS  OTHER  THAN  ZERO. 

301  IXT-XT 
T0T-C01 

IF  (XT. EQ. 0.0)  GO  TO  1001 
DO  1000  1-1 , I  XT 
R I  *  I 

P 1  *=W  1 +R I 
P2-WL-RI 
TA1-P1 
TB1-P2 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

P3-WNT+1 .0 
PA-P3-RI 
P5-RI+1 .0 

Z«  (DGAMMA  (P3) )  /  ( (DGAMMA  (Pit) )  *  (DGAMMA  (P5) ) ) 

P-TERS 

TOT-TOT+ (P*Z*W) 

1000  CONTINUE 

1001  Gl-TOT 

WRITE  (6,60)  G 1 , XT , NT 

60  FORMAT (5X, 'FIRST  CONST-' , F 10.5.5X, ' XT-' , F5- 1 ,5X, ' NT-' , I U) 
C  SO  WE  COMPUTED  THE  VALUE  OF  FIRST  CONSTRAINT 
IF  (G1.GT.ALF)  GO  TO  333 
IF  (XT. EQ. NT)  GO  TO  380 
XT-XT+1 .0 
GO  TO  301 
333  XT-XT-1.0 
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IF  (XT. LT. 0.0)  GO  TO  999 
C  OTHERWISE  WE  GO  AND  CALCULATE  G2 
380  WW«W*(DEL**WNT) 

C  NOW  COMPUTE  THE  VALUE  WHEN  XT  IS  ZERO, THAT  IS  J  IS  ZERO. 

C  WHEN  J  IS  ZERO  L  IS  ZERO 

C  WHEN  J  IS*  ZERO.M  GOES  FROM  ZERO  TO  NT  AND  L  IS  ALWAYS  ZERO  IN  THIS  CASE 
C  FIRST  CONSIDER  THE  CASE  WHERE  WHEN  M  IS  ZERO 
A=W1 
B«*W2 
T  A 1  «=W  1 
TB1-W2 

CALL  FACT2 (TA1 ,TB1 ,TERS) 

CALL  MDBETA (XI , A,B,P1 , I ER) 

CALL  MDBETA (X2, A,B,P2, I ER) 

Y=TERS 

VAL0=(P2-P1)*Y 

SUM=VALO 

C  NOW  CONSIDER  THE  CASES  WHERE  M  IS  ONE  TO  NT. 

DO  1500  M= 1 , NT 

A"=W1 

BM=M 

BMl-WNT+1.0 
BM2=WNT-BM+1 .0 
BM3=BM+1 .0 

BMCOM=DGAMMA (BM1)  / ( (DGAMMA (BM2) ) * (DGAMMA (BM3) ) ) 

BFAC=  (DEL** (-BM) ) *8 MCOM 

B=W2+BM 

TA1-W1 

TB1*B 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

CALL  MDBETA (Xl.A.B, PI, IER) 

CALL  MDBETA (X2,A,B,P2, I ER) 

Y*=TERS 

VAL«=  (P2-P1)  *Y*BFAC 
SUM=SUM+VAL 
1500  CONTINUE 
JXT*XT 
RJSUM*SUM 

C  IF  XT  IS  ZERO  WE  HAVE  ONLY  THE  ABOVE  TERM 
IF  (XT.EQ.O.O)  GO  TO  2001 
DO  2000  J-l.JXT 
C  THIS  IS  THE  MOST  OUTER  SUM 
RJ**J 

RJ1«WNT+1 .0 
RJ2-WNT-RJ+1 .0 
RJ3-RJ+1.0 

COMBJ* (DGAMMA (RJ1) )  / ( (DGAMMA (RJ2) ) *  (DGAMMA  (RJ3) ) ) 

C  NOW  L  IS  FROM  ZERO  TO  J. AGAIN  CONSIDER  THE  CASE  WHERE  L  IS  ZERO 

LP«(-l!  **J 

PL-LP 

C  NOTE  WHEN  L  IS  ZERO  M  GOES  FROM  ZERO  TO  NT-J 
LJL-NT-J 

IF  (LJL.EQ.O)  GO  TO  2101 
DO  2100  M-l.LJL 
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\% 

* 

m 


RRM-M 

RRM1-WNT-RJ+1 .0 
RRM2-WNT-R  J -RRM+ 1 .0 
RRM3-RRM+1 .0 

RC0M- (DGAMMA (RRM1) ) / ( (DGAMMA  (RRM2) ) * (D GAMMA (RRM3) ) ) 

FFAC*  (DEL**  (-RRM) ) *RCOM 
A-SGM 

B-RRM+SDEL 

TA1-A 

TB1-B 

CALL  FACT2  (TA1 ,TB1 tTERS) 

CALL  MOBETA  (XI ,A,B,P1 , IER) 

CALL  MOBETA (X2 , A , B , P2 , I ER) 

Y-TERS 

VALM= (P2-P 1) *FFAC*Y 
VALO-VALO+VALM 

2100  CONTINUE 

2101  RLSUM«VALO*PL 

C  THIS  IS  THE  VALUE  WHEN  L  IS  ZERO 

C  NOW  WE  WANT  TO  CONSIDER  L  FROM  1  TO  J.THIS  IS  THE  SECOND  SUM 
DO  2500  L«1,J 
RL-L 

RL1-RJ-RL+1 .0 
RL2-RL+1.0 

COMBL- (DGAMMA (RJ3) ) / ( (DGAMMA (RL 1) ) *  (DGAMMA (RL2) ) ) 

LPL- (- 1)  **  (J-L) 

FLP-LPL 

POWER-DEL** (-RL) 

F  ACL-F  LP*COMBL*POWER 

C  NOW  SHOULD  CONSIDER  M  LOOP  AGAIN. NOW  M  S  FROM  ZERO  TO  NT-J  FOR  GIVEN  L 
C  START  WITH  MIS  ZERO 
A-RL+SGM 
B-SDEL 

CALL  MOBETA  (X 1 ,  A, B , P 1 ,  I  ER) 

CALL  MOBETA  (X2.A.B.P2, IER) 

TA1-A 

TB1-B 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

Y-TERS 

VAL«(P2-P1)  *Y 

RMSUM-VAL 

LL-NT-J 

IF  (LL.EQ.O)  GO  TO  3001 

DO  3000  M-l.LL 

RM-M 

RM1-WNT-RJ+1.0 
RM2-WNT-RJ-RM+1.0 
RM3-RM+1 .0 

COMBM- (DGAMMA  (RM1)  )  / ( (DGAMMA (RM2) ) *  (DGAMMA  (RM3) ) ) 

FACM-  (DEL**  (-RM) )  *  (COMBM) 

A-RL+SGM 

B-RM+SDEL 

CALL  MOBETA  (XI, A, B. PI, IER) 

CALI  MOBETA  (X2, A, B,P2, IER) 
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0  r. 

I 


TA1-A 

TB1-B 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

Y-TERS 

VAL=(P2-P1)*FACM*Y 

RMSUM-RMSUM+VAL 

3000  CONTINUE 

3001  RRSUM-RMSUM 

C  THE  MOST  INNER  LOOP  IS  FINISHED. 

RLSUM- (F ACL*RRSUM) +RLSUM 
C  THIS  IS  THE  SUM  FOR  L  LOOP 
2500  CONTINUE 
C  L  LOOP  IS  F I N I  SHED 

C  NOW  FINISH  J  LOOP. THE  MOST  OUTER  LOOP. 

RJSUM- (COMBJ*RLSUM) +RJSUM 

2000  CONTINUE 

C  SO  WE  EVALUATED  G2. 

2001  G2-RJSUM*WW 

WRITE  (6.61)  G2.XT.NT 

61  FORMAT  (5X. 'SECOND  CONST-'  ,F  10.5, 5X,  'XT-'  ,F5- 1 ,5X.  'NT-',U) 

IF  (G2.LT.BET)  GO  TO  999 
777  IF  (XT.LT.l  .0)  GO  TO  888 
XT-XT-1.0 
C  CHECK  G2  AGAIN. 

WW-W* (DEL**WNT) 

C  NOW  COMPUTE  THE  VALUE  WHEN  XT  IS  ZERO, THAT  IS  J  IS  ZERO. 

C  WHEN  J  IS  ZERO  L  IS  ZERO. 

C  WHEN  J  IS  ZERO.M  GOES  FROM  ZERO  TO  NT  AND  L  IS  ALWAYS  ZERO  IN  THIS  CASE 
C  FIRST  CONSIDER  THE  CASE  WHERE  WHEN  M  IS  ZERO 
A-Wl 
B-W2 

CALL  MDBETA (X 1 , A , B , P 1 , I ER) 

CALL  MDBETA (X2.A,B,P2, IER) 

TA1-A 

TB1-B 

CALL  FACT2  (TA1 ,TB1 ,TERS) 

Y-TERS 

VALO-  (P2-P1) *Y 
SUM-VALO 

C  NOW  CONSIDER  THE  CASES  WHERE  M  IS  ONE  TO  NT. 

DO  1501  M-l.NT 
A-Wl 


J 

J 

>■ 

v* 

s' 


BM-M 

BM1-WNT+1 .0 
BM2-WNT-BM+1 .0 
BM3-BM+1 .0 

BMCOM-DGAMMA (BM1) /  ( (OGAMMA (BM2) ) * (DGAMMA (BM3) ) ) 

BFAC-  (DEL** (-BM) ) *BMC0M 

B-W2+BM 

TA1-A 

TB1-B 

CALL  FACT2  (TA1 ,TB1 ,TER$) 

CALL  MDBETA  (XI ,A,B,P1 • IER) 

CALL  MDBETA (X2, A, B,P2, IER) 


« 


Y-TERS 

VAL-  (P2-P1) *Y*BFAC 
SUM-SUM+VAL 
1501  CONTINUE 
JXT-XT 
RJSUM-SUM 

C  IF  XT  IS  ZERO  WE  HAVE  ONLY  THE  ABOVE  TERM 
IF  (XT. EQ. O.O)  GO  TO  201 1 
DO  5000  J-l.JXT 
C  THIS  IS  THE  MOST  OUTER  SUM 
RJ-J 

RJ1-WNT+1 .0 
RJ2=WNT-RJ+ 1 .0 
RJ3-RJ+1 .0 

COMBJ-  (DGAMMA  (RJ 1) )  /  ( (DGAMMA  (RJ2) )  *  (DGAMMA  (RJ3) ) ) 

C  NOW  L  IS  FROM  ZERO  TO  J. AGAIN  CONSIOER  THE  CASE  WHERE  L  IS  ZERO 
LP«(-1)**J 
PL-LP 

C  NOTE  WHEN  L  IS  ZERO  M  GOES  FROM  ZERO  TO  NT-J 
LJL-NT-J 

IF  (LJL.EQ.O)  GO  TO  2102 
DO  2105  M=1 , LJL 
RRM-M 

RRM1-WNT-RJ+1 .0 
RRM2-WNT-RJ-RRM+1 .0 
RRM3-RRM+1 .0 

RCOM- (DGAMMA (RRM1) ) / ( (DGAMMA (RRM2) ) * (DGAMMA (RRM3) ) ) 

FFAC- (DEL** (-RRM) ) *RCOM 
A-SGM 

B-RRM+SDEL 

CALL  MDBETA (XI , A, B,P 1 , I ER) 

CALL  MDBETA (X2,A.B,P2. IER) 

TA1-A 

TB1-B 

CALL  FACT2  (TA1.TB1.TERS) 

Y-TERS 

VALM-  (P2-P1) *FFAC*Y 
VALO-VALO+VALM 
2105  CONTINUE 
2102  RLSUM-VALO*PL 
C  THIS  IS  THE  VALUE  WHEN  L  IS  ZERO 

C  NOW  WANT  TO  CONSIDER  L  FROM  1  TO  J.  THIS  IS  THE  SECOND  SUM 
DO  2501  L-l.J 
RL-L 

RL1-RJ-RL+1 .0 
RL2-RL+1 .0 

COMBL- (DGAMMA (RJ 3) ) /  ( (DGAMMA (RL 1} ) * (DGAMMA  (RL2) ) ) 

LPL-(-l)**(J-L) 

FLP-LPL 

POWER-DEL** (-RL) 

FACL-FLP*COMBL*POWER 

C  NOW  SHOULD  CONSIDER  M  LOOP  AGAIN. NOW  M  IS  FROM  ZERO  TO  NT-J  FOR  GIVEN  L 
C  START  WITH  MIS  ZERO. 

A-RL+SGM 


B-SDEL 

CALL  MOBETA  (XI .A.B.P1 , I ER) 

CALL  MOBETA  (X2, A,B,P2,  l£R) 

TA1-A 

TB1-B 

CALL  FACf2 (TAl.TBl.TERS) 

Y-TERS 

VAL«(P2-P1)*Y 

RMSUM-VAL 

LL-NT-J 

IF  (LL.EQ.O)  GO  TO  1*001 

DO  4000  M- 1 , L  L 

RM-M 

RM1-WNT-RJ+1 .0 
RM2-WNT-RJ-RM+1 .0 
RM3=RM+1 .0 

COMBM- (DGAMMA (RM1) ) / ( (DGAMMA (RM2) ) * (DGAMMA (RM3) ) ) 

FACM- (DEL** (-RM) ) * (COMBM) 

A-RL+SGM 

B-RM+SDEL 

CALL  MDBETA(X1.A,B,P1.IER) 

CALL  MDBETA  (X2, A.B.P2, I ER) 

TA1-A 

TB1-B 

CALL  FACT2  (TAl.TBl.TERS) 

Y-TERS 

VAL-(P2-P1)*FACM*Y 

RMSUM-RMSUM+VAL 

4000  CONTINUE 

4001  RRSUM-RMSUM 

C  THE  MOST  INNER  LOOP  IS  FINISHED. 

RLSUM- (F  ACL*RRSUM) +RLSUM 
C  THIS  IS  THE  SUM  FOR  L  LOOP 
2501  CONTINUE 
C  L  LOOP  IS  FINISHED. 

C  NOW  FINISH  J  LOOP.  THE  MOST  OUTER  LOOP. 

RJSUM-  (COMBJ*RLSUM) +RJSUM 
5000  CONTINUE 
C  SO  WE  EVALUATED  G2. 

2011  G2-RJSUM*WW 

WRITE  (6.62)  G2.XT.NT 

62  FORMAT  (5X. 'SECOND  CONST- * F 10 .5 ,5X , ' XT- ' , F5 . 1 ,5X, ' NT- ’ , 1 4) 
C  CHECK  G2  NOW 

IF(G2.GE.BET)  GO  TO  777 
XT-XT+1 .0 
GO  TO  888 
999  NT-NT+1 
GO  TO  11 

888  WRITE (6,555)  XT. NT 

555  FORMAT (10X, 'X-' .F10.5.5X, 'N-' , 1 4) 

STOP 

END 

SUBROUTINE  FACT1  (A1 ,B1 .SON) 

IMPLICIT  REAL*8  (A-H.O-Z) 


h  *1 

:• 

i.'.'i 

if 


I5 

I 

l 


C“A1+B1 

IF  (A1 .LE. 57-0. AND. C.LE. 57-0)  GO  TO  M 

C1«C-1 .0 

A2=A1-1.0 

B2-=B  1  - 1 .0 

C2-A2+B2 

I B=A2+1 .0 

I  OC2 

PAY«C1 

DO  *»2  l-IB.IC 
Zl  =  l 

PAY=PAY*Z I 
1*2  CONTINUE 
PAYDA=1 .0 
JA=B2 

DO  1*3  J-l.JA 
VJ=J 

PAYDA«PAYDA*VJ 
1*3  CONTINUE 

SON-PAY/PAYDA 
GO  TO  A5 

1*1  SON-DGAMMA  (C)  /  ( (DGAMMA  (Al) )  *  (DGAMMA  (B 1 ) ) ) 

1*5  CONTINUE 
RETURN 
END 

SUBROUTINE  FACT2  (TA1 ,TB1 ,TERS) 

IMPLICIT  REAL*8(A-H,0-Z) 

C-TA1+TB1 

IF  (TA1.LE. 57.0. AND. C.LE. 57-0)  GO  TO  71 

Cl-C-1.0 

A2-TA1-’.  .0 

B2=TB 1-1.0 

C2-A2+B2 

I B*A2+1 .0 

IC-C2 

PAY*C 1 

DO  72  l-IB.  1C 
Z  I  =  I 

PAY«PAY*ZI 

72  CONTINUE 
PAYDA-1 .0 
JA-B2 

DO  73  J-l.JA 
VJ-J 

PAYDA-PAYDA*VJ 

73  CONTINUE 
TERS-PAYDA/PAY 
GO  TO  75 

7 1  TERS-  ( (DGAMMA  (TA1) )  *  (DGAMMA  (TB 1) ) )  /  (DGAMMA  (C) ) 
75  CONTINUE 
RETURN 
END 

//GO.SYSUB  OD 

//  DO  0SN-GWU.IMSI.V9.DL0AD.0ISP-SHR 


APPENDIX  C 


Illustrative  Calculation  of  Expected  Sample 
Sizes  for  Curtailed  Sequential  Sampling 


THE  CASE  OF  TESTING  ONE  ITEM  AT  A  TIME 

We  illustrate  this  for  Stage  0.  Here  *  5  ,  =  17. 

We  must  have  either  6  successes  to  accept,  or  12  failures  to 


P[nt=6 | Pt]  - 
Pfnt-7|,tl  - 
P[nt=8|pt]  = 
P[nt=9|pt]  = 
P[nt«10|pt]  = 
P[nt=ll|pt]  = 
P[nt«12|pt]  = 
P[nt=13|pt]  = 
P[n  =14 |p  ]  * 


r5^ 

v5. 


p“  =  0.015625 


[5  p£a-Pt)  -  0.046875 


7 

5 

(&} 

5 

'9 

5 


P^d-Pt-)2  =  0.0820312 


P^(l-Pt)3  -  0.109375 


Pt(1"Pt)4  °  0-1230469 


(5°)  p6t(l-pt)5  =  0.1230469 

fill  6.  .6  .  fill  , .  .12  . 

Is)  +  111]  <1'pt;> 

fl2l  6,,  .7  ,  fl2l  ,,  ,12 

[sJPtd-Pt)  +  |UJ  PtU-Pt) 

f  13l  6.-  .8  ^  fl3l  2,.  .12 

|  s  |  P^CI-P,)  +  Jn  ]  Pt(l~Pt) 


0.1130371 

=  0.0968018 

-  0.083313 
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P[nt-15|pt] 

P[nt=16|pt] 


P[nt=17|pt]  = 


(s4)  Pt(1“Pt)9  +  (n)  Pt<l-Pt>12  *  0.0722046 
f 55)  P^l-Pt)10  +  (“)  P^(l-Pt)12  -  0.0666504 

56]  pt(1_pt)11  +  fill  Pj(1-Pt)12  =  °*0666504 

.  /  V  A/ 


To  obtain  P[n  =j]  ,  j  =  6,7,. ..,17,  we  average  out  the  above  by 
using  g(pt|*)  .  At  Stage  0,  y  =  1  ,  6=1. 

i>iV6>  °-U28571 

p'V7!  "  6  '  °-1071429 

»tV8'  ■  21  -^(V^TbP  *  °'083333 

p[nt-9]  -  56  >  0.0666667 

p[nt»10]  -  126  -  0.0545455 

-rn  _iii  _  oco  ^ (y+6)r(6+5)  _ 
p ln^  11]  252  r(y+6+ll)  0*0454545 

p[n  =12]  =  402  I^+6)I(j±6)  +  r(Y,)_r(6-H2)  = 

Pl  t  J  T(y+6+12)  T (y+6+12)  0.1103896 

p  [„  =131  =  792  r(Y+6)r(6+7)_  r_(I±l)r(6+12)  _ 

Pl  t  ‘UJ  r(Y+6+13)  11  T(y+6+13)  0.0989011 

Tn  =1A1  _  -i 007  r(Y+6)r(6+8)  7R  r(v+2)r(6+i2)  _ 
p[  t  14^  1287  r(Y-H5+14)  +  78  T(y+6+14)  -  °-0857143 

Plnt.151  -  2002  +  364  ,  0.075 

P'V161  ■  3003  r(m^6)—  +  1365  r(r^!il)2)-  ■  °-066176 

p(pt-17]  -  4368  +  4368  r<m^'7)2>  -  0.0588235 


E[nt]  =  10.91  . 
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Pfnt-9jpt]  «=  P^(l-Pt)A  +  |3j  P^(l-Pt)A  +  ®j  P^(l-Pt)4  ■  0 .0140249 

r  ,,l  i  f9l  6„  >4  .  flO'l  1 ..  v  4  ,  fill  8M  .4  ,  fa)  10 

p[nt-12|pt]  =  [3J  Pt(l-Pt)  +[3JPt(l-Pt)  +  [  3  j  Pt(1"pt)  +  [9  J  P  t 


'lO' 


pfd-V  +  f1/]  pJ°d-Pt)2  »  0-852551 


fl2l  9, 


Pfnt=15jpt]  =  3  pt(l-pt)  -  0. 


.1291889 


Stage  2 


Pt  =  0.9  (l~Pt)  =  0.1 


x*  =  8  n  =  11 
t  t 


We  must  have  either  9  successes  to  accept  or  3  failures  to  reject, 
Thus,  nte{3,6,9,12} 


(l-pt)J  =  0.001 


P[nt=3|Pt]  = 

p[nt-6|pt]  =  [2]  Pt(l-Pt)3  +  2]  Pt(1-Pt)3  +  2)  Pt(1"Pt)3  =  °'01485 

Ptnt=9|pt]  =  [*]  P^(l-Pt)3  +  [J]  P3(l-Pt)3  +  [®]  P^l-Pt)3  +  [?]  P^  "  °- 
>  \ 

P [nt=12 1 pt ]  =  3  p^(l-pt)2  +  P^(l-Pt)  =  0.5596074 


Stage  3 


p  =  0.906  1-p  =  0.094 


x*  =  8  r.  =  11 


The  same  enumeration  as  in  Stage  2. 

Stage  4 

Pt  ■  0.909  1-p  =  0.091 


x*  =  8  nt  =  11 


4245426 


The  same  enumeration  as  in  Stage  2. 
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Stage  5 


p  -  0.875  1  -  p  -  0.125 


x*  =  9 


nt  *=  13 


The  same  enumeration  as  in  Stage  1. 


Stage  6 


p  =  0.853  1  -  p  »  0.147 


x*  =  8  n  =  12 


We  must  have  either  4  failures  to  reject  or  9  successes  to  accept. 


Thus,  nte{6,9,12} 


P[nt=6(pt]  = 


612  4  1 6 1  5  (f> 1  6 

l  P^d-Pt )  +  5  Pt(l-Pt)  +  l  d-Pt)  -  0.0054577 


p[nt=9|pt]  =  ^  P^^Pt^4  +  (3)  P4(l-Pt)4  +  [3]  P^(l-Pt)4  +  0]  pt  “  0.2653362 


p[nt=12|pt]  = 


Stage  7 


3]  P^l’Pt)3  +  [2]  Pt^-Pt>2  +  [1]  Pj(l-Pt)  =  0.7292061 


P  «  0.825  (1-p  )  -  0.175 


x*  =  9 


nt  =  14 


We  must  have  either  10  successes  to  accept  or  5  failures  to  reject. 
Thus,  nte(6,9,12,15} 

Plnt-6|pt]  =  [5]  Pt(l-Pt)5  +  [g]  (1-Pt)6  =  0.0008412 

r  ni  ,  f6)  2,,  .5  ,  (7)  3,,  .  5  f8l  4.,  .5 

Plnt-9|pt]  =  LI  Ptd-Pt)  +  4]  Pt(l-Pt)  +  4  Pt(l-Pt)  -  0.0102237 


p[nt«12|pt] 


0  P5ta-pt)5  *  («]  p°ta-pt>3  -  iv j  -  i;j  *r 

+  (“]  pfd'Pt)  +  (V)  Pt°(1-Pc>2  ’  °-6805573 

p[nt-15|pt]  =  (V]  Pt<l-Pt)4  +  (V)  P9t<l-Pt>3  *  0.3083778 


Stage  8 


p  =  0.833  (1-Pt)  =  0.167 


x*  =  9  nfc  =  14 


The  same  enumeration  as  in  Stage  7 . 
Stage  9 


p  =  0.820  (l-pt)  =  0.180 


x*  =  9  nt  =  14 


The  same  enumeration  as  in  Stage  7 . 

Stage  10 

p  =  0.837  (1-Pt)  =  0.163 


x*  =  9  nfc  ■  14 


The  same  enumeration  as  in  Stage  7. 

Stage  11 

p  =  0.841  (l-pt)  =  0.159 


x*  =10  n  -  15 


We  must  have  either  11  successes  to  accept  or  5  failures  to  reject 


Thus,  n  £{6,9,12,15} 


P[nt-6|pt]  -  [5]  Pt(l-pt)J  +  (gj  (1-Pt)°  -  0.0005289 

P [nt*9 |pt^  -  P^(l-Pt)5  +  J4J  pj(l-pt)5  +  I®  P^(l-Pt)5  “  0.0067523 

r  1  f9l  5,,  .5  .  fiol  6.,  .5  .  fill  7,.  N5  .  flOl  1 

Plnt-12|pt]  -  [4]  Ptd-Pt)  +  [4  J  Pt(l-Pt)  +  [4J  Ptd-Pt)  +  (10J  Pt 


fill  11.. 

10 J  Pt  (1'Pt 


)  =  0 . 4321114 


p [n  =15 | p  ]  = 


12 

4 


P^(l“Pt)4  + 


9  3 

P,(l-Pt)  + 


121  10  2 

Jq  p^U  (1-Pt)  «  0.5606073 


Stage  12 


p  =  0.836  (1-Pt)  =  0.164 

x*  =  9  n  ■  14 

The  same  enumeration  as  in  Stage  7. 

Stage  13. 

P  =  0.848  (l-pt)  -  0.152 

x*  =  8  n  ■  12 

The  same  enumeration  as  in  Stage  6. 

Stage  14 

Pt  =  0.850  (l-pt)  =  0.150 

x*  =  8  nfc  ■  12 

The  same  enumeration  as  in  Stage  6. 

To  obtain  the  E(nt)  ,  we  average  out  the  above  by  using  g(ptl*)  . 

We  illustrate  this  for  Stage  0. 
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& 

o 


prn  =6i  =  I<I±£L—  =  0.1428571 

plnt  J  r(y)r(6)  r(y-HS-rt) 


r  t(y-h5)  r  r(Y+6)r(<5+i)  +  ot  r(Y+6)r,(6+2i  A  ^  r(Y+6)r(.<s+3I~j  =  0.2571429 

p[nt=9]  s  r(y)f(I)  £  r(Y-HS+7)  *  21  r(Y+6+8)  w&o)  J 

T(w+6^  r..,  r(Y46)r<6+4)  .  r(Y+6)r(6+5)  /Q,  r(Y+6)XLW 

e!  V121  ■  S  L126  r(Y^t-Io)  '  +  252  r<Y*+ii)  +  402  T(y-h5+12) 


.  r(Y)r(6+12)_~|  =  0 . 2103S97 
T(Y-HS+12)  J 


r(Y+£)  nno  r(y-^)r(Y+7)  r(Y+6)r($+8)  +  2002  £(i±6)r(6+9i 

p[nt=15]  =  r(Y)T(6)  L9^  T(y+6+13)  1  r(Y-n5+14)  T(Y-h5+15) 


r(Y+l)r(6+12)  79  m+2)T(6-H_2)  r(^r±3)r(5+l2)  =  o2596154 

+  12  f(Y+6+13)  T(y+6+14)  T(Y+6+15)  J 


t(y-hS)  r —  r(Y-f5)r($+io)  .  r(Y+4)r(6+n)  ~|  =0.125 

p[nt=18]  =  p(y)r(6)  [_  T(y+<S+15)  T(Y+^+15)  J 


E[nt]  =  11-84  . 


Similarly,  we  can  obtain  E[nt]  for  other  stages. 
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